Font size
S
M
L
XL
WorksheetsBA 7
Total questions: 95
Worksheet time: 48mins
Name
Class
Date
1.
Which reason best explains why evaluation must go beyond accuracy in many business problems?
a)
Accuracy automatically accounts for unequal misclassification costs
b)
Accuracy is unaffected by class imbalance and capacity limits
c)
Decisions depend on metrics that reflect asymmetric costs and imbalance
d)
Accuracy directly measures probability calibration
e)
Accuracy is only defined for multi-class classification
2.
Which learning objective is explicitly included in the lecture?
a)
Derive gradient descent updates for logistic regression
b)
Incorporate unequal costs into decisions
c)
Build a deep neural network from scratch
d)
Perform k-means clustering and interpret centroids
e)
Estimate causal uplift with A/B tests
3.
Which item appears in the lecture agenda?
a)
Association rules and market basket analysis
b)
PR and Average Precision
c)
Text vectorization with TF-IDF
d)
Dimensionality reduction with PCA
e)
Time series decomposition
4.
In a binary confusion matrix, what does FN represent?
a)
Predicted positive when the actual class is negative
b)
Predicted negative when the actual class is positive
c)
Predicted positive when the actual class is positive
d)
Predicted negative when the actual class is negative
e)
The total number of predicted positives
5.
Which formula matches the definition of True Positive Rate (TPR), also called recall or sensitivity?
a)
TP / (TP + FP)
b)
FP / (FP + TN)
c)
TP / (TP + FN)
d)
TN / (TN + FN)
e)
(TP + TN) / (TP + FP + TN + FN)
6.
Why can a very high accuracy be misleading when positives are rare?
a)
Because accuracy ignores true negatives
b)
Because always predicting the majority class can look accurate but be useless
c)
Because accuracy cannot be computed for imbalanced datasets
d)
Because accuracy always equals precision when positives are rare
e)
Because accuracy automatically penalizes false negatives more than false positives
7.
In the F-measure family, what does the parameter beta control?
a)
The threshold used to convert scores to labels
b)
The trade-off weight between precision and recall
c)
The prevalence of the positive class
d)
The number of bins in a reliability diagram
e)
The slope of the ROC curve
8.
Which statement about the Matthews Correlation Coefficient (MCC) is correct?
a)
Its range is [0, 1], where 0 is perfect and 1 is random
b)
It is robust as a single-number metric even under class imbalance
c)
It only uses TP and FP and ignores TN and FN
d)
It always increases when prevalence decreases
e)
It is identical to accuracy for binary classification
9.
Balanced Accuracy is defined as:
a)
The geometric mean of precision and recall
b)
The arithmetic mean of sensitivity (TPR) and specificity (TNR)
c)
The harmonic mean of sensitivity and specificity
d)
The fraction of correct predictions among predicted positives
e)
The area under the ROC curve
10.
What happens to the confusion matrix when you change the decision threshold on model scores?
a)
The actual labels change but predictions stay fixed
b)
The counts TP, FP, FN, and TN can change because predicted labels change
c)
Only TP changes; FP, FN, and TN remain constant
d)
Only TN changes; TP, FP, and FN remain constant
e)
The confusion matrix is unaffected by threshold choice
11.
A ROC curve plots which pair of quantities across thresholds?
a)
Precision versus recall
b)
True Positive Rate versus False Positive Rate
c)
Accuracy versus prevalence
d)
Lift versus gain
e)
Log loss versus Brier score
12.
A ROC example table typically records TP, FP, TN, and FN at several thresholds in order to compute:
a)
Precision and recall only
b)
TPR and FPR for ROC points
c)
Brier score and ECE for calibration
d)
Lift and gain values for marketing
e)
Only accuracy values
13.
In this lecture, AUC-ROC is interpreted primarily as:
a)
A measure of probability calibration
b)
A measure of ranking ability: how often positives score higher than negatives
c)
A direct measure of expected profit
d)
A measure that depends strongly on class prevalence
e)
A metric that increases when thresholds increase
14.
On a ROC plot, an Iso-TPR (equal true positive rate) line is:
a)
A vertical line
b)
A horizontal line
c)
A diagonal line with slope 1
d)
A curve that must pass through (0, 0)
e)
A curve that depends on prevalence
15.
Which expression is equivalent to Youden's J index?
a)
Precision + NPV - 1
b)
TPR - FPR
c)
FPR - TPR
d)
TPR + FPR
e)
AUC - 0.5
16.
For highly imbalanced classification problems, the lecture recommends using which tool instead of ROC because FPR can hide false positive burden?
a)
Precision-Recall (PR) curve
b)
R-squared
c)
Mean absolute error
d)
Silhouette score
e)
Adjusted Rand index
17.
A Precision-Recall (PR) curve plots which quantities across thresholds?
a)
Precision versus recall
b)
Specificity versus prevalence
c)
TPR versus FPR
d)
Accuracy versus threshold
e)
Brier score versus log loss
18.
In PR analysis, what does the baseline precision equal?
a)
1 minus prevalence
b)
Prevalence of the positive class
c)
AUC-ROC
d)
Specificity
e)
Balanced accuracy
19.
Why is the early high-precision region of a PR curve especially valuable when review capacity is limited?
a)
It guarantees zero false negatives at any threshold
b)
It identifies operating points where you can act on a small set with few false positives
c)
It maximizes AUC-ROC by construction
d)
It makes prevalence irrelevant
e)
It ensures probabilities are calibrated
20.
Average Precision (AP) is defined in the lecture as:
a)
The unweighted mean of precision values at all thresholds
b)
A weighted mean of precision at recall increases
c)
The area under the ROC curve
d)
The maximum precision achievable at any threshold
e)
The minimum possible log loss
21.
Which statement about ROC versus PR is consistent with the lecture guidance?
a)
ROC is best only when positives are rare; PR is best only when data are balanced
b)
ROC reflects ranking ability and is prevalence-invariant; PR is preferred when positives are rare and false positives matter
c)
PR is prevalence-invariant; ROC changes only with calibration
d)
ROC directly measures probability accuracy; PR measures only costs
e)
ROC and PR always lead to identical model comparisons
22.
A model outputs a score of 0.75 for spam. If the decision threshold tau is 0.5, what is the predicted class?
a)
Not spam, because 0.75 is less than 0.5
b)
Spam, because 0.75 is greater than 0.5
c)
Spam only if the model is calibrated
d)
Not spam only if prevalence is low
e)
Cannot be determined without the ROC curve
23.
When false negatives are considered more costly than false positives, the lecture suggests tuning thresholds using which F-score setting?
a)
Choose beta < 1 to emphasize precision
b)
Choose beta > 1 to emphasize recall
c)
Choose beta = 0 to ignore recall
d)
Choose beta = 1 to ignore both precision and recall
e)
Beta has no relationship to recall emphasis
24.
One ROC-geometry method for threshold selection chooses the threshold that is:
a)
Farthest from the diagonal line
b)
Closest to the perfect point (FPR=0, TPR=1)
c)
Closest to the origin (FPR=0, TPR=0)
d)
Where precision equals prevalence
e)
Where the PR curve is flat
25.
A PR-target thresholding strategy can be described as:
a)
Maximize accuracy regardless of precision or recall
b)
Constrain precision to be at least a target value and then maximize recall
c)
Constrain AUC to exceed 0.9 and then minimize FPR
d)
Choose the threshold that makes TPR equal to FPR
e)
Always use tau = 0.5
26.
Top-k (quota-based) thresholding means you:
a)
Predict positive for all cases with score above 0.5
b)
Select exactly the k items with the highest scores
c)
Select items uniformly at random
d)
Select all items below a fixed threshold
e)
Select items only after calibration changes the ranking
27.
What does probability calibration ensure according to the lecture?
a)
Scores are always between 0 and 1
b)
A score of 0.8 corresponds to about an 80% real-world event rate among items scored 0.8
c)
The ROC curve becomes a straight line
d)
The PR curve becomes prevalence-invariant
e)
Accuracy equals balanced accuracy
28.
In the fraud-risk example, a business rule says to review any transaction with more than a 10% chance of fraud. What threshold tau implements this rule?
a)
tau = 0.9
b)
tau = 0.5
c)
tau = 0.1
d)
tau = 10
e)
tau = prevalence
29.
In cost-sensitive evaluation, a cost matrix is used to define:
a)
Only the model's hyperparameters
b)
The business value or cost for each possible prediction outcome (TP, FP, FN, TN)
c)
The order in which thresholds must be tested
d)
The number of folds in cross-validation
e)
The bins for a reliability diagram only
30.
Expected cost at a threshold is computed by:
a)
Training the model again at each threshold
b)
Computing the confusion matrix at the threshold and then applying the cost matrix
c)
Maximizing AUC-ROC and dividing by prevalence
d)
Using only true positives and ignoring other outcomes
e)
Using only calibration metrics and ignoring classification outcomes
31.
Under the simplifying assumption that correct predictions have zero cost, the Bayes-rule optimal threshold tau from costs is:
a)
tau = CFN / (CFN + CFP)
b)
tau = CFP / (CFN + CFP)
c)
tau = (CFP + CFN) / 2
d)
tau = 0.5 regardless of costs
e)
tau = prevalence of the positive class
32.
A cost curve is described in the lecture as a plot of:
a)
TPR versus FPR across thresholds
b)
Precision versus recall across thresholds
c)
Total expected cost (or profit) versus threshold
d)
Brier score versus number of bins
e)
Accuracy versus training time
33.
Risk and regulatory constraints can be incorporated into threshold choice by:
a)
Ignoring constraints and choosing the threshold with maximum AUC
b)
Adding constraints such as max FPR or min precision and solving on validation data
c)
Choosing tau on the test set to match stakeholder expectations
d)
Always using Youden's J regardless of context
e)
Replacing thresholds with clustering assignments
34.
For imbalanced data, the lecture recommends reporting which set of outputs?
a)
Only accuracy and AUC-ROC
b)
PR metrics, precision@k, and calibration information
c)
Only confusion matrix counts
d)
Only R-squared and RMSE
e)
Only p-values and confidence intervals
35.
Which pairing correctly matches a cost-sensitive approach described in the lecture?
a)
In-training: tune tau using costs; Post-hoc: apply class weights
b)
In-training: use class weights or focal loss; Post-hoc: tune tau using costs
c)
In-training: compute ROC; Post-hoc: compute PR
d)
In-training: calibrate; Post-hoc: change ranking
e)
In-training: maximize accuracy; Post-hoc: ignore costs
36.
A gain chart is used to answer which question (as stated in the lecture)?
a)
How many times better than random is the model?
b)
How many responders do we capture when targeting the top fraction of the population?
c)
What is the calibration error in the worst bin?
d)
What is the Bayes-optimal threshold from costs?
e)
How stable is the model under drift?
37.
A lift chart is used to answer which question (as stated in the lecture)?
a)
What is the probability that a positive scores higher than a negative?
b)
How many times better is our model than selecting randomly?
c)
What fraction of predicted positives are actually negative?
d)
How close is the ROC point to (0, 1)?
e)
How many bins should a reliability diagram use?
38.
Why does the lecture say probability calibration matters for business actions?
a)
Because calibration always increases AUC
b)
Because ROI and capacity decisions depend on the accuracy of predicted probabilities
c)
Because calibration eliminates the need for thresholds
d)
Because calibration guarantees zero false positives
e)
Because calibration makes prevalence irrelevant
39.
In which scenario does the lecture label calibration as critical (YES)?
a)
Ranking contests where only ordering matters
b)
Finding the top 100 users for outreach when only rank matters
c)
Calculating ROI or profit using predicted probabilities in formulas
d)
Any situation where AUC is above 0.8
e)
Any situation where the dataset is perfectly balanced
40.
Which calibration diagnostic is described as dividing predictions into bins and comparing predicted versus observed rates?
a)
ROC curve
b)
Reliability curve (reliability diagram)
c)
Lift chart
d)
Confusion matrix
e)
F1 score
41.
According to the lecture, what does the Brier score measure?
a)
Only ranking ability
b)
Mean squared error for probability predictions (calibration and discrimination)
c)
The maximum calibration error in any bin
d)
The ratio of TP to FP
e)
The slope of an iso-cost line
42.
Why does cross-validation matter for calibration according to the lecture?
a)
It guarantees the ROC curve is above the diagonal
b)
The calibrator must be trained on model predictions from data the main model did not train on
c)
It removes the need for a separate test set
d)
It forces probabilities to sum to 1
e)
It ensures accuracy is unbiased under imbalance
43.
The lecture notes that calibration almost never changes which metric because it does not change ranking?
a)
AUC-ROC
b)
Brier score
c)
ECE
d)
Log loss
e)
Precision@k
44.
When a model is uncalibrated, the lecture recommends finding an operating threshold tau* by:
a)
Assuming tau* must be 0.5
b)
Searching empirically using a cost curve on a validation set
c)
Maximizing Youden's J regardless of costs
d)
Using baseline precision as the threshold
e)
Minimizing ECE without considering costs
45.
Which workflow ordering matches the lecture's recommended evaluation process?
a)
Calibrate -> Train -> Choose threshold -> Evaluate
b)
Train -> Evaluate (ROC/PR) -> Calibrate -> Choose threshold -> Back-test
c)
Choose threshold -> Train -> Back-test -> Calibrate
d)
Evaluate -> Choose threshold -> Train -> Calibrate
e)
Back-test -> Train -> Evaluate -> Choose threshold
46.
Which operational tip is explicitly listed in the lecture?
a)
Always choose thresholds on the test set to avoid bias
b)
Track drift and recalibrate periodically
c)
Report AUC only to simplify communication
d)
Avoid stratified splits when data are imbalanced
e)
Use random splits even when prevalence shifts
47.
Which item is listed as a common pitfall in classification evaluation?
a)
Reporting AUC only
b)
Using stratified splits
c)
Comparing ROC and PR curves
d)
Running calibration diagnostics
e)
Performing back-testing
48.
When communicating results to stakeholders, the lecture recommends including:
a)
Only the single best AUC number
b)
ROC and PR plots with operating points plus cost and capacity impact for a few thresholds
c)
Only a confusion matrix at tau=0.5
d)
Only calibration metrics without any curves
e)
Only a list of hyperparameters
49.
Which item appears in the evaluation checklist?
a)
Choose tau using the test set
b)
Use ROC and PR metrics (AUC/AP) and avoid leakage
c)
Ignore capacity constraints to simplify deployment
d)
Skip calibration diagnostics when AUC is high
e)
Report only raw accuracy
50.
Which tool is mentioned in the implementation hints for computing ROC and PR curves?
a)
pandas.get_dummies
b)
sklearn.roc_curve and sklearn.precision_recall_curve
c)
statsmodels.OLS
d)
matplotlib.imshow
e)
scipy.optimize.minimize_scalar only
51.
A classifier achieves 99% accuracy on a dataset where positives are very rare, but it predicts all cases as negative. Which metric from the lecture most directly signals that this model has essentially no predictive power?
a)
AUC-ROC near 1
b)
F1 score and MCC near 0
c)
Precision near 1
d)
Specificity near 0
e)
NPV near 0
52.
If your business goal is to avoid missing true positives (false negatives are very costly), which threshold-tuning choice aligns with the F-beta guidance in the slides?
a)
Use beta < 1 so precision dominates
b)
Use beta > 1 so recall dominates
c)
Use beta = 1 so recall is ignored
d)
Do not use F-beta because it cannot be optimized
e)
Maximize accuracy instead of any F-score
53.
Two operating points have similar AUC-ROC, but one generates many more false positives. Under severe class imbalance, why can ROC look overly optimistic compared with PR?
a)
ROC uses precision on the x-axis, which hides false positives
b)
FPR can remain small even when the absolute number of false positives is large
c)
ROC is undefined when positives are rare
d)
PR is prevalence-invariant while ROC depends on prevalence
e)
ROC automatically incorporates misclassification costs
54.
You have a fixed review capacity and can only investigate the top 2% highest-risk cases. Which evaluation approach is the most appropriate match to the lecture?
a)
Choose tau to maximize AUC-ROC and then review all positives
b)
Use top-k / quota-based selection and report precision@k and recall@k
c)
Set tau = prevalence to match the baseline
d)
Only report balanced accuracy because it is a single number
e)
Only report ECE because calibration is all that matters
55.
AUC is described as P(score_pos > score_neg). Which statement follows from this interpretation?
a)
AUC measures probability accuracy and changes strongly after calibration
b)
AUC depends on the chosen threshold tau
c)
AUC reflects ranking quality and is unaffected by monotonic score transforms
d)
AUC is the same as balanced accuracy under class imbalance
e)
AUC increases when prevalence increases, holding scores fixed
56.
Your use case is medical diagnosis where the predicted probability enters a treatment decision and a cost calculation. According to the slides, what is the strongest reason you should care about calibration?
a)
Calibration will always increase AUC-ROC
b)
Calibration makes the confusion matrix independent of thresholds
c)
The probability value itself must be accurate because it is used directly in decisions and ROI/cost formulas
d)
Calibration eliminates false positives at any threshold
e)
Calibration is only needed when data are balanced
57.
Suppose a model's reliability curve has an 'S' shape indicating over-confidence. After calibration, which change is most consistent with the slides?
a)
AUC-ROC should increase substantially
b)
The curve should bend closer to the diagonal and ECE should decrease
c)
Precision should become equal to prevalence at all thresholds
d)
The ROC curve should become a straight diagonal line
e)
Balanced accuracy should become exactly 1
58.
If you calibrate a model but do not change its ranking of instances, which metric is expected to stay approximately the same?
a)
AUC-ROC
b)
Brier score
c)
ECE
d)
Log loss
e)
Reliability curve shape
59.
You must meet a regulatory requirement of max FPR while minimizing expected cost. Which approach best matches the lecture's recommendation?
a)
Choose the threshold that maximizes Youden's J regardless of constraints
b)
Solve a constrained optimization on validation data: minimize cost subject to the FPR constraint
c)
Choose the threshold on the test set to meet the constraint
d)
Always use tau = 0.5 to satisfy regulations
e)
Use only PR-AUC because constraints cannot be applied to thresholds
60.
A team reports only AUC-ROC for a highly imbalanced fraud problem and claims the model is ready. Which missing element is most critical per the slides?
a)
A larger font size on the ROC plot
b)
PR metrics and an operating point tied to capacity/cost constraints
c)
A proof that AUC equals MCC
d)
Removing the confusion matrix from the report
e)
Replacing thresholds with clustering
61.
Baseline precision in PR analysis equals prevalence. What implication does this have when positives are extremely rare?
a)
A random classifier can still have high baseline precision
b)
Even modest precision values may represent large gains over baseline
c)
PR curves are unaffected by prevalence
d)
ROC and PR curves become identical
e)
AP must equal 0.5 for any model
62.
Average Precision (AP) emphasizes early retrieval quality. Which operating scenario best aligns with that emphasis?
a)
You will act on almost every case, so late-recall behavior is most important
b)
You will only act on the highest-scored cases due to limited capacity
c)
Your costs are perfectly symmetric and prevalence is 50%
d)
You only care about true negatives
e)
You only care about calibration and not ranking
63.
If you increase the decision threshold tau, which directional change is most likely, holding score rankings fixed?
a)
More predicted positives, typically increasing recall
b)
Fewer predicted positives, typically decreasing recall and reducing false positives
c)
No change to TP, FP, FN, TN
d)
Precision must decrease while recall increases
e)
AUC-ROC must decrease because tau changed
64.
Youden's J is recommended as an 'optimal' balanced choice under which condition stated in the lecture?
a)
When costs are highly unequal and prevalence is extremely low
b)
When costs are roughly equal and prevalence is around 50%
c)
When probabilities are uncalibrated
d)
When only rank matters (top-k selection)
e)
When regulatory constraints forbid threshold tuning
65.
A company uses an uncalibrated model where a score of 0.1 corresponds to about 30% real-world risk. If they set tau = 0.1 to follow a 10% risk rule, what is the most likely outcome?
a)
They will exactly meet the 10% review rule in the real world
b)
They will systematically under-trigger reviews and miss many true positives
c)
They will trigger reviews at a much higher real-world risk level than intended
d)
They will automatically fix calibration by changing tau
e)
They will preserve the same cost curve shape but shift AUC upward
66.
Under the Bayes-rule threshold tau = CFP / (CFN + CFP) with correct-prediction costs set to zero, what happens to the optimal threshold when the cost of a false negative increases while CFP stays fixed?
a)
tau increases, making positives harder to predict
b)
tau decreases, making positives easier to predict
c)
tau stays the same because it depends only on CFP
d)
tau becomes 0.5 regardless of costs
e)
tau becomes equal to prevalence
67.
The slides describe cost curves as plotting total expected cost across thresholds. How would you use such a curve in practice?
a)
Pick the threshold where the curve reaches its minimum on validation data
b)
Pick the threshold where AUC-ROC is maximized
c)
Pick the threshold where precision equals recall
d)
Pick the threshold where the ROC curve crosses the diagonal
e)
Pick the threshold where ECE is maximal
68.
Which statement best distinguishes cost-sensitive learning from post-hoc cost-sensitive evaluation?
a)
Cost-sensitive learning changes the evaluation metric; post-hoc changes the dataset
b)
Cost-sensitive learning incorporates costs during training (e.g., weights); post-hoc tunes the threshold using costs after training
c)
Cost-sensitive learning is only for ROC; post-hoc is only for PR
d)
Cost-sensitive learning requires calibration; post-hoc forbids it
e)
There is no difference; they are the same approach
69.
A gain chart and a lift chart answer different questions. Which pairing is correct per the lecture?
a)
Gain: 'How many times better than random?'; Lift: 'How many responders do we capture?'
b)
Gain: 'How many responders do we capture?'; Lift: 'How many times better than random?'
c)
Gain: 'How calibrated are probabilities?'; Lift: 'What is AUC?'
d)
Gain: 'What is FPR?'; Lift: 'What is TPR?'
e)
Gain: 'What is log loss?'; Lift: 'What is Brier score?'
70.
In the lecture's checklist mindset, which combination best supports a defensible threshold choice for deployment?
a)
Pick tau on the test set and report only AUC-ROC
b)
Use stratified splits, avoid leakage, compare ROC and PR, and choose tau under cost/capacity constraints
c)
Maximize accuracy and skip calibration diagnostics
d)
Use only reliability diagrams and ignore PR/ROC
e)
Use only MCC and ignore business costs
71.
If your goal is a marketing campaign for a very expensive product, which PR-target thresholding choice aligns with the slide examples?
a)
Set precision >= 0.9 and then maximize recall
b)
Set recall >= 0.9 and then maximize precision
c)
Maximize Youden's J
d)
Minimize Euclidean distance to (0, 1) on ROC
e)
Always use top-k with k equal to all customers
72.
A model is excellent for ranking but poorly calibrated. Which situation from the slides is most likely to be harmed by miscalibration?
a)
Ranking contest where only ordering matters
b)
Finding the top 100 users when only rank matters
c)
Setting insurance premiums based on predicted risk probabilities
d)
Reporting ROC curves in a classroom
e)
Comparing two models with identical AUC
73.
Which calibration metric is described as penalizing confident errors heavily ('surprise' level)?
a)
Balanced accuracy
b)
Log loss (cross-entropy)
c)
MCC
d)
AP
e)
Youden's J
74.
To train a calibrator without leaking information, the slides recommend:
a)
Training the calibrator on the same data used to train the main model
b)
Using a hold-out set for the calibrator or out-of-fold predictions, then evaluating on untouched data
c)
Choosing tau on the test set and then fitting the calibrator
d)
Skipping cross-validation and using random splits
e)
Calibrating only after deployment drift is detected
75.
Which statement is most consistent with the slide 'When ROC Misleads' about unequal misclassification costs?
a)
ROC already encodes different error costs through its axes
b)
ROC treats all errors equally; cost-benefit analysis or weighted errors are needed
c)
PR curves cannot be used when costs are unequal
d)
Cost-sensitive analysis is only relevant when data are balanced
e)
Unequal costs affect AUC but not threshold choice
76.
A team picks a threshold that maximizes F1 but uses the test set to do it. Why is this flagged as a pitfall in the slides?
a)
Because F1 cannot be computed from a confusion matrix
b)
Because it creates leakage and leads to overly optimistic evaluation
c)
Because ROC curves require exactly 10 thresholds
d)
Because calibration requires unstratified splits
e)
Because Youden's J must always be used instead
77.
After deploying a model in production, what action is recommended in the practical tips to maintain probability quality over time?
a)
Never revisit calibration once the model is shipped
b)
Track drift and recalibrate periodically
c)
Stop reporting PR curves
d)
Replace cost curves with accuracy curves
e)
Increase tau each month regardless of performance
78.
You are screening for a rare condition with extremely limited follow-up capacity. Missing a true case is very costly, but you can only act on a small number of top-scored cases. Which evaluation summary best fits the slide guidance for this situation?
a)
Focus on AUC-ROC only, because it is prevalence-invariant
b)
Focus on early PR behavior (AP) plus precision@k for the actionable top fraction
c)
Focus on raw accuracy, because capacity is limited
d)
Focus on specificity only, because positives are rare
e)
Focus on calibration only, because ranking does not matter
79.
Two models have nearly identical AUC-ROC. Model A has noticeably higher AP and much better precision in the highest-score region. Under a fixed manual-review quota, which conclusion is most defensible from the slides?
a)
Prefer Model A because the early high-precision region is valuable under capacity constraints
b)
Prefer Model B because AUC is the only valid metric under imbalance
c)
The models are equivalent because AUC is identical, regardless of PR behavior
d)
Prefer the model with lower prevalence because AP is prevalence-invariant
e)
Prefer the model with higher accuracy because quotas replace curves
80.
Costs are CFP = 10 and CFN = 90 (correct predictions have zero cost). Using the Bayes-rule threshold tau = CFP / (CFN + CFP), which threshold policy is implied?
a)
A high threshold near 0.9 to avoid false positives
b)
A low threshold near 0.1 to reduce false negatives
c)
A threshold of exactly 0.5 because costs cancel out
d)
A threshold equal to prevalence
e)
No thresholding is needed; only ranking matters
81.
A team maximizes Youden's J to choose tau for an application where false negatives are far more costly than false positives and prevalence is below 1%. Based on the slides, what is the main issue with this choice?
a)
Youden's J is defined only for multi-class problems
b)
Youden's J assumes roughly equal costs and a near-balanced prevalence, so it may be economically suboptimal here
c)
Youden's J cannot be computed without calibration
d)
Youden's J always minimizes expected cost regardless of costs
e)
Youden's J is identical to AP, so PR should be avoided
82.
Which statement best captures the interaction between calibration and thresholding described in the slides?
a)
Calibration usually changes the ranking, so AUC must change
b)
Calibration rarely changes AUC because ranking is unchanged, but it can materially change the profit-optimal threshold interpretation
c)
Calibration removes the need to choose a threshold because outputs become labels
d)
Calibration makes ROC and PR curves identical
e)
Calibration guarantees that tau = 0.5 is optimal under any costs
83.
A churn model is used to compute expected ROI where the ROI formula multiplies offer cost by predicted churn probability. The model is over-confident (systematically high probabilities). Which failure mode is most consistent with the slides?
a)
Only the rank order is wrong, but ROI remains correct
b)
The business may overspend on expensive retention offers because the probabilities are inflated
c)
AUC-ROC will necessarily fall below 0.5
d)
Precision will necessarily equal prevalence
e)
Calibration error will be zero by definition
84.
You know CFP and CFN, but your model scores are not calibrated. According to the slides, which approach is the most appropriate way to find an operating threshold?
a)
Compute tau directly from costs and deploy it as a real risk cutoff
b)
Use an empirical cost curve on validation data to search for the cost-minimizing threshold
c)
Maximize AUC-ROC because it incorporates costs implicitly
d)
Set tau to prevalence because it matches the PR baseline
e)
Pick the point closest to (0, 1) on ROC because it is always cost-optimal
85.
You must satisfy both (1) max FPR and (2) min precision while also minimizing expected cost. Which procedure aligns with the slide guidance without relying on visual inspection of plots?
a)
Scan thresholds on validation data, keep only those meeting both constraints, then choose the one with minimum expected cost
b)
Choose the threshold that maximizes accuracy, then check constraints afterwards
c)
Choose the threshold that maximizes Youden's J, because it enforces constraints automatically
d)
Choose tau = 0.5 and report calibration metrics instead
e)
Choose the threshold that maximizes AP, because it guarantees both constraints
86.
Which calibration training pipeline best avoids leakage, given the slide guidance on cross-validation for calibration?
a)
Train the main model on all data; train the calibrator on the same data and predictions
b)
Generate out-of-fold predictions for the training set; train the calibrator on those predictions; evaluate final calibration on an untouched set
c)
Train the calibrator on test-set predictions to maximize calibration accuracy
d)
Train the calibrator only on positives to reduce imbalance
e)
Skip a calibrator and tune tau on the test set instead
87.
In a top-k selection workflow, which statement is the most accurate implication from the slides?
a)
The numeric value of the threshold score must correspond to a real probability, otherwise ranking is impossible
b)
Ranking quality and precision@k are central, but calibration becomes critical only if probabilities are used in downstream calculations
c)
Calibration always matters more than ranking because k is fixed
d)
AUC is invalid because it depends on k
e)
PR curves cannot be used with top-k selection
88.
The slides note that two ROC-similar points can differ economically. Which additional artifact most directly resolves that ambiguity for decision-making?
a)
A bigger ROC plot with more thresholds
b)
A cost-benefit analysis or expected-cost table for candidate thresholds
c)
A single accuracy number at tau=0.5
d)
A list of model hyperparameters
e)
A calibration metric only (e.g., ECE) with no operating points
89.
You need to decide whether to optimize for coverage of responders or efficiency relative to random selection. Which pairing of chart and objective is correct per the slides?
a)
Lift chart for coverage; Gain chart for efficiency
b)
Gain chart for coverage; Lift chart for efficiency
c)
ROC curve for coverage; PR curve for efficiency
d)
Reliability diagram for coverage; Cost curve for efficiency
e)
MCC for coverage; F1 for efficiency
90.
A reliability diagram uses bins of predicted probability. The slides mention that counts per bin provide stability context. What is the key risk if you use too many bins with limited data?
a)
AUC-ROC will become undefined
b)
Observed rates per bin become noisy, making calibration assessment unstable
c)
Precision becomes equal to recall automatically
d)
The confusion matrix stops depending on tau
e)
PR-AUC becomes larger than 1
91.
Two models have similar Brier scores, but Model X has much worse log loss. Using the slide definitions, what is the most plausible interpretation?
a)
Model X makes more confident wrong predictions, which log loss penalizes heavily
b)
Model X has better calibration because log loss prefers larger values
c)
Model X must have higher AUC because log loss measures ranking
d)
Model X must have lower prevalence because Brier depends only on prevalence
e)
This situation cannot happen if both models are calibrated
92.
The practical tips mention fixing SRM before evaluation. In experimentation terms, what is the primary reason this is important for downstream threshold and ROI decisions?
a)
SRM increases the number of bins in a reliability diagram
b)
SRM can bias estimated outcomes and costs, leading to incorrect operating-point and ROI conclusions
c)
SRM guarantees that AUC increases
d)
SRM makes PR curves prevalence-invariant
e)
SRM forces tau to be 0.5
93.
Stakeholders ask for a 'what-if' analysis under prevalence shifts. Based on the slides, which metric baseline is most directly affected by prevalence and should be interpreted accordingly?
a)
AUC-ROC baseline
b)
PR baseline precision
c)
MCC range endpoints
d)
Youden's J definition
e)
The ROC diagonal line slope
94.
A team wants to reflect a very high cost of false positives on a ROC plot. Which concept from the slides provides the correct geometric intuition for how costs relate to ROC threshold choice?
a)
Choosing the point with maximum vertical distance from the diagonal
b)
Using iso-cost line slope to select an optimal operating point on ROC when costs are unequal
c)
Using baseline precision as a horizontal reference line
d)
Using reliability diagrams to pick the best ROC point
e)
Using lift charts to define TPR and FPR
95.
The slides say teams often combine cost-sensitive learning and post-hoc threshold tuning. Which rationale best supports using both?
a)
Training-time weighting can improve learning under imbalance, while post-hoc tuning adapts the operating point to business costs and constraints
b)
Both methods are identical, so using both doubles the AUC
c)
Post-hoc tuning changes model weights, so it must be combined with training
d)
Cost-sensitive learning is only for calibrated models, while post-hoc is only for uncalibrated models
e)
Using both guarantees that Youden's J is maximized
Reset
