How can credal sets quantify epistemic uncertainty in conformal prediction?

Set size collapses a whole profile into one count

Conformal prediction attaches a p-value to every candidate label for a test input, and the prediction set at error level α collects the labels whose p-value exceeds α. The paper calls the full vector of these p-values the conformal profile or contour. Prediction-set size |Cα(x)| reads only the labels above one threshold, so in a K-class problem it can take at most K+1 distinct values.

The paper's motivating figure makes the loss concrete: two inputs can have identical prediction-set size at α = 0.05 even though one has smaller p-values for every competing label. Because the contour determines how the set changes as α varies, the authors ask whether the complete profile can produce finer and more useful uncertainty rankings than a single-level count.

This matters for decisions that depend on ordering rather than on a single cut, such as which predictions to abstain from or which observations to flag as potentially shifted. A score that saturates at a handful of values cannot separate inputs that share a set size.

A normalised profile defines a set of compatible distributions

The paper works in the framework of imprecise probability, where uncertainty is represented by a credal set: a nonempty, closed, convex family of probability distributions over the label space. Its upper probability assigns each event the largest probability any member of the family gives it, and the core is the set of distributions whose event probabilities stay below those upper bounds.

The central representational claim is that a suitably normalised conformal profile induces such a core for split conformal prediction, extending a correspondence previously identified for full conformal prediction. The nested prediction sets obtained by varying the threshold are also linked to imprecise highest density regions, which generalise classical highest density regions from one distribution to a family.

A credal set does not yield a single scalar uncertainty. The paper therefore studies three families of summaries: nonspecificity, measured by Generalised Hartley over the core's Möbius coefficients; imprecision in entropy, comparing entropies across the distributions in the set; and upper–lower expectation gaps. Each captures a different aspect of what the core encodes.

Exact computation replaces exponential enumeration

Computing these measures naively would require enumerating exponentially many events over subsets of labels. The paper derives analytical characterisations and exact algorithms for the established credal measures in the conformal case, reducing the computation to linear time after sorting the conformal p-values.

The authors also unify several credal measures with classical conformal summaries by integrating set-size penalties based on Tsallis entropies along the path of nested prediction sets. This makes the weighting behind different summaries explicit: the penalty determines how much influence leading candidates versus lower-ranked labels receive.

That unification is not merely cosmetic. In the experiments, changing the nonconformity score reverses which penalty performs better, which the authors attribute to the different emphasis each penalty places on the highest contour ranks versus lower-ranked labels.

The scores summarise total conformal uncertainty

The uncertainty literature usually separates aleatoric uncertainty, arising from randomness in the target, from epistemic uncertainty, arising from limited knowledge of the predictive mechanism. The paper explicitly interprets its scores as summaries of total conformal uncertainty: they reflect conditional label variability, model estimation, and finite calibration data together, as captured by the chosen nonconformity score.

Separating those components is stated as beyond the scope of the work. A controlled experiment supports the interpretation: in a correctly specified three-class softmax model with least-ambiguous-classifier scores, increasing training and calibration samples to 30,000 each removes sampling variation but does not drive conformal ambiguity to zero. The joint estimates were 0.1625, 0.6740, and 0.1881 for the total-variation gap, entropy imprecision, and Generalised Hartley, against oracle references of 0.1629, 0.6753, and 0.1886, a largest absolute discrepancy of 0.0014.

Ambiguity decreased with mean conditional entropy and was exactly zero for a Dirac-label population. So in this controlled setting, a non-degenerate population contour leaves a positive ambiguity floor even with abundant data; the scores are not a pure measure of estimation ignorance.

Full-profile summaries beat fixed-level cardinality

In selective prediction, the task is to rank inputs by decreasing uncertainty and measure accuracy on the retained predictions, integrated over 0–90% rejection and reported as normalised AUARC. Using a fixed pretrained CIFAR-10 ResNet-56, the Generalised Hartley summary and the saturating Tsallis summary both exceeded every tested fixed-level cardinality on all four datasets. Paired gains over the strongest mean cardinality ranged from 0.85 to 1.86 percentage points for Generalised Hartley and 0.50 to 1.80 for the saturating summary.

Against probability-vector baselines the picture is mixed. Relative to the strongest of the three probability baselines in each condition, Generalised Hartley gained 0.135±0.031 points on Caltech-101 and 1.196±0.071 on ImageNetV2, while essentially matching entropy on CIFAR-100. Under the least-ambiguous-classifier score, the saturating summary exceeded all three probability baselines on Caltech-101 and ImageNetV2 with gains of 0.077±0.017 and 0.684±0.101 points, but Generalised Hartley retained an advantage on the five conditions with at least 100 labels.

For distribution-shift detection, the protocol calibrates on 5,000 clean CIFAR-10 examples and evaluates balanced clean/shifted detection with AUROC, treating shifted examples as positive. Generalised Hartley exceeded predictive entropy on all six aggregate benchmarks, in every paired seed, and on the mean of each of the 75 corruption conditions, with paired gains of 0.305±0.020 points over entropy and 0.335±0.025 over the second-largest probability on the corruption average. This was not uniform dominance: on CIFAR-10.1, Generalised Hartley trailed the second-largest probability by 0.047±0.038 points.

The authors note a protocol limitation: corrupted observations are sampled independently from the full cache of 10,000 base images, so a corrupted view of an image used for clean calibration can appear in the shifted evaluation sample. The results should not be read as evaluation on entirely unseen base images, though the same sampling rule applies to all four nonconformity scores.

Regression rank cores cannot separate inputs

The paper proves an impossibility theorem for common split-conformal regression constructions: under the conditions of the theorem, the conformal rank core is identical across inputs, so no fixed uncertainty functional of it can distinguish them, even when prediction regions vary in response space.

This is a structural limit rather than a sample-size problem. It means the credal-geometry programme as formulated for classification does not automatically transfer to regression: the projected rank core discards the response geometry that differs between inputs. The authors suggest that recovering discrimination would require summaries that retain relevant response geometry, or conformal constructions whose rank cores vary across inputs, and that such extensions should state which changes in response location or scale matter for the intended decision.

The scope condition is explicit: the theorem holds under the stated assumptions for common split-conformal regression constructions. It is not a claim that no credal summary of any conformal regression method can ever rank inputs.

The best summary depends on score and task

The paper's own take-home is narrower than a claim of superiority. Credal summaries of the full profile yield substantially better uncertainty rankings than fixed-level cardinality in the main least-ambiguous-classifier experiments, but their relative performance against summaries of the original probability vector depends on the task, the fitted model, and the nonconformity score.

The score dependence is visible in the ablation: under the adaptive prediction sets score, the saturating summary improved on Generalised Hartley in all seven condition means, reversing the ordering seen under the least-ambiguous-classifier score. Under the smaller-adaptive-prediction-sets variant, all four core summaries fell below the three probability baselines on every aggregate, a result the authors tie to the fixed weight, randomisation, calibration size, and classifier tested.

A practical reading follows from these comparisons rather than from a general rule. When the decision needs to separate inputs that share a set size at a fixed error level, the full-profile credal summaries are the paper's demonstrated alternative, and Generalised Hartley is the stronger default under the least-ambiguous-classifier score on large-label conditions. When the question is whether to trust a credal summary over a probability-vector summary, the paper's evidence says to check the specific nonconformity score and dataset first, because the ordering can reverse. And for split-conformal regression under the theorem's conditions, no fixed functional of the rank core will rank inputs at all.

Sources used in this answer