Why do four-stage sleep agreement rates for the same consumer ring vary so widely across validation studies?

The yardstick is itself wobbly, especially for light sleep

A validation study works by scoring the same night twice: once by polysomnography, the clinical standard, and once by the ring, then counting how often the two agree on each 30-second block. That sounds like a clean comparison, but the clinical side is produced by humans following visual rules, and humans do not fully agree with each other. In one multicentre test, ten scorers from seven sleep centres each scored the same 50 polysomnography recordings. Overall agreement was reasonably strong, but the scorers matched each other perfectly on only 48% of blocks, and agreement was lowest for N1, the light transitional stage, at a kappa of 0.41 — a level usually described as fair rather than good [3].

This matters directly for four-stage agreement. The four-stage scheme lumps light NREM sleep together, and light sleep is exactly where human scorers are least reliable [3]. When the reference label is itself uncertain, a ring that disagrees with one scorer may agree with another. Part of the spread across studies is therefore not the ring behaving differently but the benchmark shifting underneath it. The same study found agreement also dropped as the rate of sleep-stage transitions rose and as breathing-pause severity increased, meaning two validation samples with different mixes of fragmented sleepers will not produce the same number even with identical hardware [3].

Different ring generations and different sleepers produce different numbers

Agreement figures are often quoted as if they describe "the Oura Ring," but the device has changed across generations, and reviews of the field note substantial variability between devices alongside fair-to-moderate agreement with polysomnography — four-stage agreement values reported in the range of 0.21 to 0.53 [4]. A range that wide, spanning roughly poor to moderate, is a signal that the measurement conditions differ, not that one study is simply right and another wrong.

The people being measured differ too. A ring validated on healthy volunteers in a home setting is not being tested under the same conditions as one validated in a sleep-laboratory population, and a laboratory sample enriched for sleep disorders will contain more fragmented nights — the very condition associated with lower scoring agreement [3][5]. So a headline agreement rate carries its population with it. A number from a clinical sample and a number from healthy home users are answering slightly different questions, and comparing them directly overstates the disagreement between studies.

The ring infers sleep rather than measuring brain waves

The ring does not record brain activity. It reads motion and signals from the autonomic nervous system — the body's automatic regulation of heart rate and related functions — plus circadian timing cues, and a model turns those into stage labels [1]. In a large dataset of 440 nights from 106 people, a motion-only model reached 57% four-stage accuracy, while adding the autonomic and circadian features raised it to 79% [1]. That gap is instructive: much of what the ring knows about sleep comes from indirect bodily signals, so its performance depends on how strongly those signals track brain-state changes in the particular people wearing it.

This also explains why two-stage and four-stage results should never be mixed. The same model reached 94–96% accuracy for the simpler sleep-versus-wake question but only 57–79% for the four-stage version [1]. A study that reports a high figure may be describing the easier binary task, and a reader comparing it with a four-stage figure will see a discrepancy that is really a difference in what was scored. One analysis argues that binary sleep–wake accuracy is inflated by base rate — because most of the night is sleep, a device can score well by mostly saying "sleep" — which makes the four-stage number the more demanding and more variable test [5].

How to read an agreement number when you see one

Treat any single four-stage agreement figure as a statement about a specific ring generation, a specific group of sleepers, and a specific scoring protocol, not as a property of the device in general [4][5]. The most useful question is not "which study is correct" but "what was this number measured against, and in whom." A study on a sleep-laboratory population with many disordered sleepers is testing a harder case than one on healthy home users, and its lower number is not necessarily a worse device [3][5].

It also helps to know that some of the variation is irreducible. As long as four-stage scoring depends on human judgment of ambiguous light-sleep blocks, even two perfect devices would show different agreement rates against two different scorer panels [3]. The wide spread across validation studies is best read as a combination of genuinely different hardware and populations and a reference standard that carries its own disagreement — which is why a range, rather than a single number, is the honest summary of how well a consumer ring stages sleep [4].

Sources used in this answer

Related research questions