quiz week 12

Conformal Prediction (CP) ensures marginal coverage but struggles with conditional coverage. Which solutions address this issue?

• Bayesian Conformal prediction

• Studentized Conformal prediction

• Quantile

  Conformal prediction

• Mondrian

  Conformal prediction

• Boosted Conformal prediction

Answer explanation

Studentized, Quantile, and Mondrian Conformal Prediction methods enhance conditional coverage, addressing the limitations of standard CP, which primarily ensures marginal coverage.

What is the result of the dotted Area?

• 1-alpha

• 1+alpha

• alpha

• The value of the Critical Score

Answer explanation

It is 1-alpha because rapresents the cumolative probability of the scores S that are less than or equal to the critical value.

Why are Mondrian Conformal Predictions applicable only when X is discrete?

• Because they require a natural partition of the space

  X into distinct subgroups.

• Because they are designed to reduce computational effort on discrete datasets.

• Because they do not guarantee marginal coverage on continuous data

• Because they cannot handle dependencies among continuous variables

Answer explanation

Standard Conformal Predictions guarantee only marginal coverage, meaning that the predictive interval contains the true value with an average probability of 1−α, calculated over the entire dataset.

However, this does not imply that the coverage is uniform across all subpopulations or specific conditions (X=x). In some regions of the feature space, the coverage may be greater or less than 1−α.

Mondrian Conformal Predictions aim to overcome this limitation by dividing the dataset into distinct subgroups, ensuring that conditional coverage is valid within each subgroup

What is the length of the Jump in an equally weighted conformal prediction?

• 1/n

• n/(1+n)

• 1/(n+1)

• (n+1)/(1-n)

Answer explanation

In equally weighted conformal predictions, the conformity scores (S_1,S_2,…,S n) are ordered, and each observation is assigned the same weight. The resulting empirical distribution has n+1 intervals, including n scores and one additional space to account for the tail beyond the last score. The jump in the empirical distribution between two consecutive scores is given by:

1/n+1 where n is the total number of observations. This value reflects the fact that the distribution assigns uniform weight to each of the n+1 intervals.

• The critical point cannot be calculated.

• The prediction intervals perfectly adapt to the local characteristics of the data.

• The prediction intervals become static and correspond to those of standard conformal predictions.

• The coverage probability decreases to less than 1−α

Answer explanation

When gamma is equal to 0 the intervals become static and uniform, exactly following the behavior of the standard CP

In adaptive conformal predictions, what should you do if many errors occur (i.e., the true value falls outside the prediction interval)?

• Increase

  α to obtain larger prediction intervals.

• Decrease

  α to widen the prediction intervals and improve coverage.

• Decrease the adaptivity parameter γ to stabilize the prediction intervals

• Increase the size of the training data to recalibrate the conformity scores.

Answer explanation

Reducing the value of α in conformal predictions increases the required confidence level (1−α). This leads to Wider prediction intervals and Improved coverage

In Studentized Conformal Predictions, how is the conformity score normalized to account for local variability?

• By dividing each conformity score by a local estimate of the standard deviation.

• By using the uniform distribution to scale the conformity scores.

• By applying a weight inversely proportional to the score's value.

• By averaging the conformity scores within each data subgroup.

Answer explanation

This normalization allows the method to adapt to the local variability of the data, ensuring that the scores are comparable even in regions with different dispersion. As a result, Studentized CP improves conditional coverage compared to standard conformal predictions.

• They are the normalized scores used to calculate conditional coverage.

• They are the model prediction and the true target value, respectively.

• They are the critical thresholds that separate the scores between training and test data.

• They are the estimated quantiles of the model residuals that define the lower and upper bounds of the prediction interval.

Answer explanation

In the context of Quantile Conformal Predictions, the values of Tau represent the estimated quantiles of the model's residuals, which are used to determine the lower and upper bounds of the predictive interval.

What are the main steps (we saw in class) to follow to build predictive intervals with Adaptive Conformal Predictions?

• 1)Calculate the conformity scores.

  2)Normalize the scores using a local estimate of variability.

  3) Sort the scores and determine the critical point.

• 1)Construct a predictive interval using the classical method of conformal prediction.

  2)Check if the empirical coverage meets the desired confidence level.

  3)Adapt the interval if the empirical coverage is insufficient.

• 1)Divide the feature space into distinct subgroups.

  2)Calculate the conformity scores for each subgroup.

  3)Use the scores to construct separate predictive intervals.

• 1)Estimate the conditional quantiles of the target distribution.

  2) Build predictive intervals around the estimated quantiles.

  3) Calculate the marginal coverage over the entire dataset.

Answer explanation

In Adaptive Conformal Predictions, the process begins by constructing a predictive interval using the classical conformal prediction method. Then, the empirical coverage of the interval is checked to ensure it meets the desired confidence level. If the coverage is insufficient, the interval is adjusted to improve coverage without compromising too much on precision. This adaptive process allows for the improvement of the quality of predictive intervals based on the specific data.

In Adaptive Conformal Predictions, which statement is correct regarding the expectation of errors?

• The expectation of errors is exactly equal to α

• The expectation of errors is adapted for each subset of data, and thus depends on the chosen confidence level.

• The expectation of errors is less than or equal to

  α.

• The expectation of errors is greater than α to obtain wider predictive intervals.

Answer explanation

The expectation of errors must be less than or equal to α, meaning the probability that a predictive interval does not contain the true value (error) must be controlled and not exceed the desired confidence level α. This ensures that the intervals provide adequate predictive coverage.