How can selective sleep staging accept more epochs at a fixed risk target?

Coverage is bought with an error budget

A sleep-staging cascade lets a small primary classifier keep the epochs it handles reliably and pass the rest to a stronger model. The first stage is therefore a selective-classification problem: accept as many epochs as possible, measured as coverage, while keeping the error rate among accepted epochs, called accepted risk, under a prescribed target. COVER studies only this first stage for a fixed primary classifier; the paper states that a complete second stage remains future work.

The distinction matters because the two quantities trade off directly. Raising coverage admits more uncertain epochs, and the accepted-risk constraint is what stops the acceptance set from absorbing confidently wrong predictions. COVER's evaluation uses a 5% accepted-risk target on Sleep-EDF-20, with subjects held out from all fitting and selection, so the reported coverage is measured on subjects the scorer never saw during development.

Equal confidence can hide different errors

Maximum softmax probability (MSP) is the standard acceptance signal: epochs whose top class probability is high are accepted. The paper's argument against relying on it is that equal MSP values can conceal different runner-up stages, different distribution shapes, or different agreement with an auxiliary model, and those differences may carry information about primary errors that a single confidence value discards.

COVER's first step, auxiliary-informed primary-error learning, builds a 37-feature risk vector from the outputs of two models: an EEG Conformer as the primary classifier and a random forest on handcrafted features as the auxiliary model. Fourteen features summarize the primary probabilities, fourteen summarize the auxiliary probabilities, and nine capture disagreement, including forward, reverse and symmetric KL divergence, Jensen-Shannon divergence, label disagreement, and the probability each model assigns to the other's predicted class. Disagreement is paired with both models' confidence because disagreement alone is ambiguous: either model may be the wrong one. The released label always stays the primary prediction.

Two objectives, two different jobs

The initial error score is a subject-weighted logistic regression trained to predict whether the primary classifier erred, using binary cross-entropy. That objective rewards error prediction on every epoch, but first-stage utility depends only on the epochs that are accepted, so a good average error predictor need not produce the largest low-risk acceptance set.

Fixed-scale refinement closes that gap by optimizing a smooth version of the coverage objective itself: a sigmoid gate replaces the hard accept-or-reject decision, giving differentiable coverage and accepted risk, and the refinement maximizes soft coverage subject to soft risk staying at or below the target. The paper explains the practical difference through gradients: binary cross-entropy weights every epoch by its residual, while refinement scales each epoch's contribution by a term that peaks at the soft acceptance boundary and is small in saturated regions. Refinement therefore spends the scorer's limited linear capacity on epochs whose acceptance is genuinely in question.

The fixed-scale part addresses a confound. The gate depends on the threshold, the weights and the temperature only through their ratio, so scaling all three together leaves every gate unchanged while changing the regularization penalty. COVER anchors the scale to the binary cross-entropy solution, fixes the temperature at 0.2, and keeps the scale and intercept fixed during refinement so the regularization strength acts on a consistent reference.

The learned score accepts far more epochs

The headline comparison isolates the first step. Auxiliary-informed error learning raises mean subject coverage from 31.5% for MSP to 48.8% at similar subject-equal risk. That is a coverage comparison at a matched risk level, not a claim that the learned score is better calibrated.

A second comparison holds acceptance volume fixed instead. When MSP accepts the same number of epochs as the learned scorer in each subject, 20,639 epochs in total, errors fall from 1,467 to 867. This matched-budget view answers a different question from the coverage comparison: given the same amount of work deferred to a later model, how many accepted labels are wrong. The paper reports that the linear binary cross-entropy scorer has fewer matched errors than MSP for 16 of 20 subjects.

Refinement adds a small but consistent gain

Fixed-scale refinement adds 1.7 percentage points of coverage over its initialization in nested development. In the main comparison across eight evaluated scorers, COVER attains the highest mean coverage, 50.4% at 4.5% subject-equal risk, above the selective-ranking baseline SELE at 49.3% and the probability-fusion comparator DuoF at 43.6%.

The margin over SELE is narrow, and the paper is explicit about what that does and does not show. Within the 30-60% acceptance region, the saved Linear-BCE, FixedScale and SELE curves differ by at most 0.41 percentage points, and the authors note that similar quota performance does not establish a uniform ranking advantage because locked rules can allocate acceptance differently across subjects. The matched-budget risk difference between FixedScale and Linear-BCE is +0.11 percentage points with a descriptive interval of -0.18 to +0.48, which spans zero.

Acceptance is set on subjects the scorer never saw

COVER separates three subject roles in each outer fold. FIT subjects develop the scorer, CERT subjects lock the acceptance rule with the scorer frozen, and OUTER subjects evaluate the frozen system. Risk vectors for FIT subjects come from out-of-fold component predictions, so the scorer learns from outputs of the kind it will see at deployment, where components score subjects they were not trained on.

The acceptance rule is chosen on CERT by maximizing realized subject coverage subject to empirical accepted risk at or below the target, with a support constraint requiring at least 50 accepted epochs from every CERT subject. That support constraint is not a per-subject risk bound. If no candidate rule is feasible, COVER abstains on all epochs. The paper describes this as empirical rule selection without a finite-sample guarantee.

One cohort, one architecture, one seed

The reported results come from a single 20-person cohort, one primary architecture and one protocol seed. FixedScale was specified after a prespecified multi-condition campaign that did not meet its success gate, and the Linear-BCE anchor used for the main matched comparisons was chosen after the prespecified analysis. Nested development reuses frozen component outputs, which limits independence at the component level.

The paper also notes that the auxiliary random forest's input features do not enter the risk vector, so COVER and DuoF draw on outputs of the same two components; the comparison therefore differs in loss, capacity and selection rather than in available information. No downstream model is evaluated, so error resolution and end-to-end computational savings, including auxiliary and scoring costs, remain unmeasured.

The error score carries the gain, not the refinement

Read together, the results assign the two innovations very different weights. The move from MSP to an auxiliary-informed learned error score accounts for the large change in coverage, 31.5% to 48.8% at similar subject-equal risk, and for the drop in matched-budget errors from 1,467 to 867. Fixed-scale refinement contributes 1.7 percentage points over its initialization in nested development, and its matched-budget risk difference from Linear-BCE spans zero.

That ordering is the paper's most decision-relevant message for anyone building a cascade: the acceptance signal, not the refinement objective, is where the coverage comes from. The refinement's stated role is to improve the locked operating point over its binary cross-entropy initialization, and the authors describe the similar matched-budget rankings and riskier exchanged sets as consistent with the boundary-focused objective without establishing a uniform ranking advantage. A next evaluation would need to freeze the complete scoring and selection pipeline before applying it to previously unexposed subjects, and a downstream model would be needed to convert first-stage coverage into measured end-to-end savings.

Sources used in this answer