Does neuropsychologically calibrated relabeling improve speech-based cognitive assessment?

Why self-reported labels limit speech-based screening

Automatic detection of cognitive impairment from conversational speech is attractive because it is remote and non-invasive, but the paper identifies a data problem that sits upstream of any model. The CognoMemory platform collects speech from UK memory clinics, where diagnoses are clinically verified, and also from volunteers recruited through websites and social media, where participants supply their own diagnostic label. Those self-reported labels are not checked by a clinician involved in data collection, and the paper notes they may reflect outdated diagnoses or a participant's limited awareness of their own cognitive status.

Training a classifier on such labels means teaching it to reproduce uncertain supervision. The study's central claim is that this label noise, rather than the choice of classifier, is the critical bottleneck for speech-based cognitive assessment. The test of that claim is whether rebuilding the labels from objective cognitive measurements improves predictions on participants whose diagnoses were clinically confirmed.

Rebuilding labels from memory, attention and executive scores

The study analysed 138 hours of conversational speech from 720 participants. Instead of treating the original annotations as fixed, it reassigned each participant to dementia, mild cognitive impairment (MCI) or healthy control using their scores on three neuropsychological domains: memory, attention and executive function. Two cut-offs, T1 and T2, separate the three categories, and participants scoring at or below T1 were labelled dementia, those between T1 and T2 MCI, and those above T2 healthy control.

Three ways of placing those cut-offs were compared. The FLAME strategy borrowed diagnostic boundaries from an external clinical cohort by taking midpoints between that cohort's group means. The Cogno Gaussian strategy estimated the same kind of midpoints from a clinically validated subset of CognoMemory. The Cogno z-score strategy instead standardised each score against the healthy-control mean and standard deviation, labelling scores below −2.5 standard deviations as dementia and scores between −2.5 and −1.5 as MCI. In a second variant, available MoCA scores overrode the domain-based assignment, with MoCA ≤17 as dementia, 18–25 as MCI and >25 as healthy control.

The paper describes these reassigned labels as clinically grounded proxy annotations rather than formal diagnoses. Their purpose is to supply cleaner supervision for the speech models, and the thresholds were deliberately kept moderately conservative to avoid the very small groups that extreme cut-offs would create.

Memory scores held up across recruitment sources

The relabeling strategy depends on the cognitive scores meaning the same thing regardless of where a participant was recruited. The paper reports that attention accuracy and executive function scores showed several statistically significant differences between clinical and non-clinical sources across the dementia, MCI and unknown groups. Memory scores, by contrast, were largely non-significant across sources.

That difference matters for how much weight each domain can carry. Because memory performance was stable across acquisition contexts, the authors treat memory-based thresholds as the most suitable basis for large-scale semi-automatic dataset curation. The same comparison also showed that memory-based reassignment produced the most stable class distributions, while attention- and executive-based thresholds varied more depending on which calibration strategy was used.

The reassignment itself shifted the dataset's composition. Original annotations were strongly imbalanced toward MCI, whereas all strategies redistributed participants toward healthier categories, producing proportions closer to the external test cohort. Z-score thresholding gave the most conservative impairment estimates, assigning fewer dementia labels and enlarging the healthy-control group.

Binary F1 rose from 0.70 to 0.83

Transformer-based classifiers (BART, RoBERTa and DistilBERT) were fine-tuned on the original labels and on each set of reassigned labels, then tested on a held-out set of 150 participants who had all been recruited through clinical pathways and therefore had clinically confirmed diagnoses. Because the data are imbalanced, macro F1 is the primary metric, and a majority-voting step combined predictions across question combinations.

On this internal test set, memory- and MoCA-based z-score calibration produced the greatest improvements. Binary classification F1 rose from 0.70 to 0.83, a 13% increase, and multiclass F1 rose from 0.54 to 0.66, a 12% increase. The paper identifies the Cogno z-score memory approach as the strongest overall performer.

The gap between the two tasks is itself informative. Multiclass classification requires separating MCI from both dementia and healthy control, an intermediate state the paper describes as harder to distinguish, which is why its absolute F1 remains lower even after relabeling.

External gains were real but much smaller

The same memory-based calibration was also evaluated on three external corpora: Pitt, ADReSS and PROCESS-2. There the increases were +4%, +5% and +9% respectively. These are the same direction of effect as the internal result but on a different scale, and the paper does not report them as equivalent to the internal +13% and +12%.

Reading the two sets of numbers together sets the boundary of the claim. The large internal gains were measured on 150 clinically verified participants drawn from the same platform as the training data, while the external corpora are separate datasets with their own recording and recruitment conditions. The consistent but smaller external improvements support the idea that better labels help beyond the original cohort, but they do not reproduce the internal magnitude.

Relabeling also changed the linguistic signal

Beyond the F1 numbers, the paper reports a TF-IDF analysis of word use across diagnostic groups. Participants labelled dementia relied predominantly on highly prototypical category exemplars, whereas healthy controls showed broader semantic exploration. The authors interpret this as evidence that neuropsychologically informed labelling surfaces clinically interpretable linguistic markers rather than only improving classifier optimisation.

That interpretation belongs to the authors, and it rests on the reassigned groups rather than on independent clinical diagnosis. It is nonetheless the paper's answer to why the relabeling helps: the new labels group participants in a way that aligns with measurable differences in how they speak, not just with who reported which diagnosis.

Label quality outranks architecture in this comparison

The study's practical priority follows from its comparison structure. The classifiers were held to transformer-based architectures across conditions, so the performance differences reported here come from changing the supervision, not the model. The authors conclude that improving diagnostic label quality produced larger gains than modifying model architecture, and they frame data curation as the primary bottleneck in automatic cognitive decline detection.

The most decision-relevant split in the paper is therefore between label sources and domains. Memory-based z-score calibration, optionally stabilised by MoCA priors, gave the best results internally and the only external improvements reported; attention and executive thresholds were more sensitive to recruitment source and calibration choice. For anyone building speech-based screening on real-world data with unverified annotations, that ordering, memory first and clinical priors where available, is the paper's specific recommendation rather than a general call for bigger models.

Sources used in this answer