Intro to ML: Evaluation (Part 1)

Our model predicts which people have Covid-19. The model correctly predicts 100 people who have Covid-19, but it misses 50 people who have the illness and falsely predicts that 10 people have the illness when they don't. What is the precision of the model?

• 91%

• 66%

• Not enough information

Answer explanation

The precision of a model is calculated as the number of true positives (100 in this case) divided by the sum of true positives and false positives (100+10). This gives us 100/110 = 0.909, which is approximately 91%. Therefore, the correct answer is 91%.

A diagnostic test is used to identify individuals with a certain disease. The test correctly identifies 120 individuals with the disease, but fails to identify 30 individuals who actually have it, and incorrectly identifies 15 individuals as having the disease. What is the recall of the test?

• 80%

• 75%

• Not enough information

Answer explanation

Recall is calculated as true positives divided by the sum of true positives and false negatives. Here, recall = 120 / (120 + 30) = 120 / 150 = 0.8 or 80%. Thus, the correct answer is 80%.

If we predict True for every observation, what will our model recall be?

• Not enough information

• 0%

• 100%

Answer explanation

The recall of a model is calculated as the number of true positives divided by the sum of true positives and false negatives. If we predict True for every observation, there will be no false negatives, hence the recall will be 100%. This is because we are correctly identifying all positive cases, even though we may also be incorrectly identifying some negative cases as positive.

Which of the following do we use a test set for?

• We use it to evaluate new features that we could add to our model

• We use it for hyper-parameter tuning

• We use it for held-out performance evaluation

• None of the above

Answer explanation

The test set is used for held-out performance evaluation. This means it is used to assess the performance of a model on unseen data, providing an unbiased evaluation of the final model fit. It is not used for hyper-parameter tuning or evaluating new features for the model.

When is cross validation particularly helpful?

• When we have a small dataset

• When we have a large dataset

• When we are particularly constrained on computational resources

Answer explanation

Cross validation is particularly helpful when we have a small dataset because it allows us to maximize the use of limited data for both training and validation. By dividing the dataset into multiple folds and iteratively training and validating on different subsets, we can obtain a more accurate estimate of the model's performance and reduce the risk of overfitting.

Which of the following does an F1 score depend on?

• TP, FP, FN

• TP, TN, FP, TN

• TN, FP, TP

• TP, FN

Answer explanation

The F1 score is a measure of a test's accuracy and it depends on True Positive (TP), False Positive (FP), and False Negative (FN). It is the harmonic mean of precision and recall, where precision is TP/(TP+FP) and recall is TP/(TP+FN). Therefore, the correct answer is 'TP, FP, FN'.

Which of the following could you use to evaluate performance on a regression task?

• Precision

• Recall

• Accuracy

• MSE

Answer explanation

MSE, or Mean Squared Error, is the correct choice for evaluating performance on a regression task. It measures the average of the squares of the errors—that is, the average squared difference between the estimated values and the actual value. Precision, Recall, and Accuracy are metrics used for classification tasks, not regression tasks.

What is the accuracy of a model that correctly predicts 150 cases out of 200 total cases, with 30 false positives?

• 75%

• 85%

• Not enough information

Answer explanation

Accuracy is calculated as (True Positives + True Negatives) / Total Cases. Here, (True Positives + True Negatives) = 150, Total Cases = 200. Thus, Accuracy = 150/200 = 0.75 or 75%. Therefore, the correct answer is 75%.

Which metric is most appropriate for evaluating a model when the cost of false negatives is very high?

• Precision

• Recall

• F1 Score

• Accuracy

Answer explanation

Recall is the most appropriate metric when false negatives are costly, as it measures the ability of the model to identify all relevant instances. High recall ensures that most actual positives are correctly predicted.

In a classification task, if a model has a high precision but low recall, what does this indicate?

• The model is good at identifying all positive cases

• The model is good at identifying positive cases but misses many

• The model is not good at identifying positive cases

• The model has a balanced performance

Answer explanation

High precision means the model correctly identifies many positive cases, but low recall indicates it misses a significant number of them. Thus, the model is good at identifying positive cases but misses many.