How do test-retest reliability problems in EEG markers affect reproducibility of cognitive impairment classification?

Why a stable EEG marker is necessary for reproducible classification

A classification is only as reproducible as the measurements it rests on. If the same person's EEG marker fluctuates from one test session to the next, then a cutoff that separates impaired from healthy will place that person on different sides of the line at different times—not because their cognition changed, but because the measurement did. This is the practical consequence of poor test-retest reliability: the label becomes partly a function of when the test was taken.

In a longitudinal study of event-related potentials (ERPs), which are averaged EEG responses time-locked to a cognitive task, the reliability of different ERP components was not uniform. The late positive potential (LPP) amplitude showed high reliability in healthy controls, with intraclass correlation coefficients (ICCs) of 0.85 for target stimuli and 0.75 for non-target stimuli. The P200 latency was only moderately reliable, with ICCs of 0.67 and 0.53 for target and non-target stimuli, respectively. [1] An ICC is a number between 0 and 1 that describes how much of the variation between measurements is due to real differences between people rather than measurement noise; higher values mean more stable markers.

Reliability is marker-specific, so classification reproducibility varies by marker

Because reliability differs across ERP components, the reproducibility of a classification rule will also differ depending on which component it uses. A rule based on LPP amplitude, which was highly reliable in controls, would be expected to produce more consistent labels over time than a rule based on P200 latency, which was only moderately reliable. [1] This means that a study reporting good classification accuracy at a single time point does not guarantee that the same classification will hold at a later test.

The same study also found that individuals with mild cognitive impairment (MCI) showed a detectable increase in P200 latency at one-year follow-up, with a large effect size. [1] That finding indicates the marker is sensitive to change, but sensitivity and reliability are different properties: a marker can detect group-level change while still being noisy enough at the individual level to make a single person's classification unstable. The paper reports reliability only for healthy controls, so the stability of these ERP measures within the MCI group itself is not established by this evidence.

Analysis choices can add a second layer of irreproducibility

Even a marker with acceptable test-retest reliability can yield inconsistent classifications when researchers make different preprocessing or analysis decisions. A multiverse analysis of the EEG theta/alpha ratio—a spectral measure sometimes used as a marker of cognitive impairment—examined how alternative processing and analysis choices affect the reproducibility and stability of findings. [2] The implication is that two labs could start from the same raw EEG and reach different conclusions about who is impaired, not because the underlying biology differs but because the analysis pipeline differs.

This specification dependence compounds the problem of measurement noise. If a marker is only moderately reliable to begin with, and analysis choices further shift the values, the combined uncertainty can be large enough to change individual classifications. The multiverse study frames this as a challenge for standardized, reproducible evaluation of quantitative EEG markers of cognitive impairment. [2] It does not quantify how much of the classification instability comes from test-retest noise versus analysis choices, so the relative contribution of each remains an open question.

What this means for using EEG to track or classify impairment

For longitudinal monitoring and treatment trials, the practical answer is to prefer markers with demonstrated high test-retest reliability and to report reliability alongside classification accuracy. The ERP study supports the feasibility of longitudinal monitoring because its most reliable measure, LPP amplitude, remained stable over a year in healthy controls. [1] But feasibility for group-level monitoring is not the same as guaranteed reproducibility for individual classification, especially for moderately reliable measures like P200 latency.

A study protocol for Alzheimer's disease and MCI biomarkers includes test-retest reliability assessment of EEG changes as part of its design. [3] That inclusion reflects a recognition that reliability needs to be measured, not assumed. However, the protocol does not itself analyze how reliability issues affect the reproducibility of cognitive impairment classification, so it does not resolve the question. The safest conclusion from the available evidence is that reliability problems can degrade classification reproducibility, that the degree of degradation depends on the specific marker and analysis pipeline, and that markers should be selected and reported with their reliability in mind. [1][2]

Sources used in this answer

Related research questions