Video-only staging asks a different question
Most recent video-based sleep staging systems treat the camera as a route to something else. They reconstruct respiratory or cardiac waveforms, derive optical-flow motion features, or align video with EEG-guided cross-modal representations. Those designs answer how well video can recover or imitate a physiological signal.
This paper asks whether the video itself, with no proxy reconstruction and no auxiliary physiological supervision, contains enough evidence to assign sleep stages under labels defined by polysomnography. That distinction matters because blankets, position changes and face anonymization can make respiratory or rPPG extraction unreliable, and a one-dimensional proxy may discard regional motion patterns and brief movements.
The setting is not a replacement for PSG. Certified technicians scored the reference stages from EEG, EOG and EMG under AASM guidelines; the model only tries to reproduce those labels from video.
Static scenes hide weak, localized motion
Standard video models are built for actions, objects and scene changes, where the discriminative cue is usually large motion or semantic interaction. In overnight near-infrared sleep video the scene is mostly static, and useful evidence appears as weak local micro-motion.
The paper shows what goes wrong with a baseline 3D CNN on raw video. Frame-difference and spatial attention maps highlight static high-contrast regions rather than temporally coherent thoraco-abdominal motion, and a t-SNE projection of epoch embeddings from six subjects clusters mainly by subject, with stages dispersed inside each subject.
Subject-specific factors such as body shape, bedding, camera geometry and baseline respiration-related motion can therefore dominate the representation. The interpretation of motion also depends on where it occurs: weak thoraco-abdominal periodicity, fast limb movement and broad position changes carry different stage evidence.
Encoding micro-motion before spatial reduction
ViNUSS, short for Video-Native Unmediated Sleep Staging, is built around three components. The first is spatially anchored pre-spatial micro-motion encoding, which preserves localized temporal variation together with its spatial context before spatial processing compresses the signal.
The ablation isolates its contribution. Starting from a vanilla MoViNet-A2 backbone at 0.60 accuracy and 0.50 macro-F1, adding this encoding alone raises the scores to 0.62 accuracy and 0.55 macro-F1. The gain is modest but consistent with the claim that weak local temporal cues survive better when they are encoded before spatial reduction.
Backbone choice was tested separately. Video Swin-Tiny reached 0.51 accuracy and 0.46 macro-F1, MoViNet-A5 reached 0.60 and 0.53, and MoViNet-A2 reached 0.60 and 0.56 with roughly 4.0M parameters. The larger MoViNet-A5 has more than four times the parameters of A2 yet did not improve accuracy and had lower macro-F1, which is why A2 was selected.
Stage cues measured against each night's baseline
The second component, within-subject stage contrast, learns stage cues relative to each subject's night-specific baseline rather than to a population average. This targets the subject-dominant clustering seen in the baseline embeddings.
On its own the effect is small: adding it to the micro-motion encoding variant moved accuracy from 0.62 to 0.60 while macro-F1 rose from 0.55 to 0.56. The authors read this as improved stage-wise separability under subject-specific variation rather than a broad accuracy gain.
Its clearer role appears in temporal coherence. Measuring cosine similarity between consecutive epoch embeddings, the paper reports that within-subject stage contrast increases this coherence across stages, producing smoother feature trajectories that the full-night model can use.
Full-night dynamics carry the largest single gain
The third component, two-scale sleep dynamics modeling, captures within-epoch motion evolution and organizes epoch-level evidence into a full-night trajectory. A short-range path summarizes motion inside each 30-second epoch, and a long-range path models the stacked epoch tokens across the night.
Adding this dynamics module to the vanilla backbone produced the largest single jump in the ablation: macro-F1 rose from 0.50 to 0.67 and accuracy from 0.60 to 0.75. The benefit depends on representation quality, however. With the same dynamics module, adding micro-motion encoding raised macro-F1 to 0.72, and adding both encoding and subject-relative contrast reached 0.78.
The sequence-model comparison supports the two-scale design. Among models using only inter-epoch context, a long-range Mamba path reached 0.78 accuracy and 0.74 macro-F1, ahead of BiLSTM, PatchTST, ModernTCN and a vanilla Transformer. The full two-scale design improved every class over that long-only version, with Deep F1 rising from 0.59 to 0.67.
Reading 0.80 accuracy against the right baseline
On the held-out test set, ViNUSS reaches 0.80 accuracy, 0.68 Cohen's kappa and 0.78 macro-F1 for four-class staging, where N1 and N2 are merged into Light and N3 is treated as Deep. The same trained model scores 0.71 accuracy and 0.72 macro-F1 when evaluated with all five AASM labels, so the two figures come from one model with different label mappings, not from two separate systems.
Class balance matters for interpreting these numbers. In the four-class confusion matrix, Deep has the weakest per-class precision at 0.67 and recall at 0.68, while Light reaches 0.83 F1 and REM 0.82. Macro-F1 is emphasized because stages are imbalanced and minority stages such as Deep and REM are clinically important.
The comparison with a proxy-first method needs care. Reproducing the fully video-based SleepVST pipeline on this dataset was not feasible because its video-to-waveform module is not public and face anonymization plus bedding occlusion make rPPG and respiratory reconstruction unreliable. The authors instead evaluated oracle-input variants of that architecture, one with contact-sensor cardiac waveforms and one with both respiratory and cardiac waveforms. Against the full-oracle variant, ViNUSS improved macro-F1 from 0.73 to 0.78. These are oracle-input versions of one representative architecture, not universal upper bounds for physiological-signal-based staging.
Saliency points to breathing and body movement
Sleep Video EigenCAM was built to inspect which spatiotemporal regions the model uses, separating temporal and spatial channel weights before combining them so that motion-related and appearance-related structure are not mixed.
In a stable Deep sleep epoch, saliency concentrates around the thoraco-abdominal region and the temporal saliency curve follows respiration-related periodic motion. In an epoch containing brief arousals and a position change, saliency highlights the larger movement when it occurs and returns to the thoraco-abdominal region during quieter intervals.
The authors interpret this as evidence that the model uses physiologically meaningful visual motion without explicitly reconstructing a respiratory waveform. That is an interpretability observation about attention patterns, not a demonstration that the model measures respiration.
Where the result holds and where it stops
The evaluation covers 475 overnight near-infrared recordings totaling about 3,250 hours from the Korea Video Sleep Study, captured at 5 fps with a wall-mounted camera under routine sleep-study conditions including far-field viewpoints, bedding occlusion, weak motion and face anonymization. Subjects were split into 335 training, 70 validation and 70 test recordings, stratified by sex, age and OSA severity.
A stratified analysis reports per-subject accuracy across blanket occlusion, camera angle, room re-entry, light intensity, sex, age group, BMI class and AHI group, with no clear systematic degradation across the observed subgroups. That supports stability within this cohort and these acquisition conditions.
The authors state that broader validation across additional clinical sites and home environments remains important. The practical priority follows from the comparison inside the paper: video-only staging is most valuable where proxy extraction is fragile, such as occluded or anonymized recordings, and its 0.78 macro-F1 against a 0.73 full-oracle proxy-first variant is the specific margin a reader should weigh, not a claim of clinical replacement.
Sources used in this answer
Subject-Relative Micro-Motion and Sleep Dynamics for Near-Infrared Video Sleep Staging
ViNUSS stages sleep from near-infrared video alone, without physiological proxy reconstruction, reaching 0.80 accuracy and 0.78 macro-F1 for four-class staging on 475 overnight recordings scored against PSG.