Intro to ML: Decision Trees - 2024

Which system requires more entropy to be described?

• Tossing a fair coin

• Tossing a biased coin

Answer explanation

Tossing a fair coin requires more entropy to be described because it has a higher level of uncertainty. In a fair coin toss, both outcomes (heads and tails) have an equal probability of occurring, making it more difficult to predict the outcome. In contrast, a biased coin has a higher probability for one outcome, reducing the overall uncertainty and entropy.

Is it possible to test the same attribute twice along the same path on a decision tree for a categorical problem?

• Yes

• No

Answer explanation

No, it is not possible to test the same attribute twice along the same path on a decision tree for a categorical problem. Once an attribute is used for a split, it should not be used again in the same path, as it would not provide any additional information or improve the classification.

Is it possible that the same attribute will get selected twice in an ordinal or real-valued problem?

• Yes

• No

Answer explanation

In ordinal or real-valued problems, it is indeed possible for the same attribute to be selected twice. This can occur due to the nature of these problems, where the same attribute may have different impacts at different stages or conditions. Therefore, the answer to the question is 'Yes'.

Decision trees are an algorithm for which machine learning task?

• Clustering

• Classification

• Classification and Regression

• Dimensionality reduction

• Regression

Answer explanation

Decision trees are a versatile machine learning algorithm that can be used for both classification and regression tasks. They work by creating a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. Therefore, the correct answer is 'Classification and Regression'.

Oliver is analysing a decision tree model for predicting the types of plants in a garden. He notices that the tree is significantly deep. What does this indicate?

• The samples have a large number of attributes

• The dataset is possibly noisy

• The tree under-fits the training data

• None of these

Answer explanation

A significantly deep tree indicates that the dataset is possibly noisy. This is because a deep tree tends to overfit the data, capturing noise and making it less generalizable to new data. A large number of attributes or underfitting the training data are not directly related to the depth of the tree.

Which of the following is true:

• Deeper trees will always improve performance on the training data

• Deeper trees will always improve performance when testing the model on unseen data

• If a deeper tree improves performance on the training data, then it will also improve performance on new unseen data

Answer explanation

Deeper trees will always improve performance on the training data because they can capture more complex patterns and relationships in the data. However, this does not guarantee better performance on unseen data, as deeper trees can also lead to overfitting, which reduces the model's ability to generalize to new data.

Kiara is working on a binary classification problem to predict whether a customer will make a purchase or not. She has a balanced dataset, and she finds that the customer's age completely determines the class for each observation. What is the information gain from using the age feature as the first node in a decision tree?

• 0

• 0.5

• 1

• Not enough information

Answer explanation

In a binary classification problem with a balanced dataset, if one feature completely determines the class for each observation, the information gain from using this feature as the first node in a tree is 1. This is because the feature perfectly partitions the data, resulting in pure child nodes with no uncertainty remaining.

How does a Decision Tree decide which feature to split on at each node?

• By selecting the feature with the highest variance

• By selecting the feature with the most unique values

• By selecting the feature that results in the purest child nodes (highest information gain or lowest impurity)

• By selecting features randomly

Answer explanation

A decision tree splits on the feature that results in the purest child nodes. This is achieved by maximizing the information gain or minimizing the impurity.

What is overfitting in the context of Decision Trees?

• It occurs when the tree is too shallow

• It occurs when the tree is too deep and captures noise in the data

• It is not possible in Decision Trees

• It is a measure of the tree's purity

Answer explanation

Overfitting in Decision Trees occurs when the tree is too deep, causing it to capture noise in the data. This happens because the tree becomes too complex and starts fitting to random variations in the data, rather than the underlying patterns. As a result, the model performs poorly on new, unseen data.

What is pruning in the context of Decision Trees?

• Trimming the tree by removing branches that do not contribute to classification accuracy

• Adding more nodes to increase tree depth

• Reducing the number of features in the dataset

• Changing the root node of the tree

Answer explanation

Pruning in Decision Trees refers to the process of trimming the tree by removing branches that do not contribute to classification accuracy. This is done to prevent overfitting and improve the model's ability to generalize.