Why the fluency index resists simple word counts
The Edinburgh Cognitive and Behavioural ALS Screen (ECAS) is a multi-domain screen for cognitive and behavioural impairment in people with ALS, the most common form of motor neuron disease. Its verbal fluency subtest asks a person to say as many words as possible beginning with a target letter in one minute, excluding proper nouns. Raw word counts are problematic here because slowed speech can look like executive dysfunction. The Verbal Fluency Index (VFI) was designed to separate the two: it is an articulation-speed-adjusted estimate of the average time taken to think of each word.
Computing it is labour-intensive. A clinician first runs the generation phase, then a controlled reading phase in which the participant reads the accepted words aloud; the clinician removes repetitions, hesitations and proper nouns and times the reading. The resulting VFI is normalised against predefined thresholds into a verbal fluency score from 0 to 12. The paper's motivation is that automating this from audio could reduce clinician supervision, support remote assessment and lower burden on patients and the health service.
How the pipeline reconstructs word timing
The system does not simply transcribe speech. It needs precise boundaries around each correctly produced word and the gaps between words, because the VFI depends on timing rather than content alone. The authors compared several speech recognition systems, including Wav2Vec2, HuBERT, NVIDIA NeMo, Whisper and WhisperX, and report that WhisperX produced significantly more accurate word-level timestamps, largely through phoneme-level forced alignment. They therefore report results using WhisperX.
For speech versus pause boundaries they tested Silero, FSMN and CNN-BiLSTM voice activity detection, scoring segmentation with four error types (speech false alarms, speech misses, pause false alarms and pause misses) summed and normalised by total audio duration. Voice activity detection alone gives only speech-pause boundaries, which the authors say are inadequate for their features, so adjusted boundaries are required. Text processing with SpaCy and NLTK stemming supports correct-word identification, repetition detection and filler-word flagging. The paper notes that dysarthric speech causes recognition errors such as "pod" for "part", and that background noise makes robust speech detection necessary.
Clinical timing features beat acoustic and embedding baselines
The authors compared three feature families: traditional acoustic descriptors such as eGeMAPS, spectral and voice-quality measures; self-supervised embeddings from Wav2Vec2 and HuBERT; and clinically interpretable features such as correct word count, average word duration, pause ratio and disfluency counts. They trained SVR, XGBoost, random forest and LASSO models, plus deep regression heads, and evaluated with 5-fold cross-validation using R² and normalised root mean squared error (NRMSE). Classical models consistently outperformed the deep learning counterparts, which the authors attribute to the small dataset.
The gap was large. The best baseline, an SVR on HuBERT layer 8 embeddings, reached R² 0.526 and NRMSE 0.102 for P-words, and R² 0.353 and NRMSE 0.164 for S-words. The fully automatic clinical-feature pipeline reached R² 0.856 and NRMSE 0.053 for P-words with LASSO, and R² 0.832 and NRMSE 0.083 for S-words. The authors also ran an oracle condition using clinical features derived from manual alignments as an upper bound; LASSO there reached R² 0.797 and NRMSE 0.064 for P-words and R² 0.865 and NRMSE 0.066 for S-words. A paired t-test over 10×5-fold cross-validation with different random seeds found no statistically significant difference between models trained on manual and automatic clinical features for either target letter.
The generation phase alone may be enough
The standard ECAS fluency assessment uses S-words, but this study finds P-words yield comparable estimation performance. More consequentially, the authors report that estimation is achievable using only the generation phase, which would remove the controlled reading phase and its timing from the workflow. That claim matters because the reading phase is one of the steps that makes manual VFI scoring slow and dependent on clinician supervision.
The comparison should be read with its conditions attached. The P-word and S-word results come from the same 42 participants and the same cross-validation scheme, and the abstract's headline figures (P-words R² 0.9, NRMSE 0.05; S-words R² 0.8, NRMSE 0.08) are rounded summaries of the table values above. The paper's own framing is feasibility: it demonstrates that accurate regression models can be built, not that the index has been validated for clinical decision-making.
Dysarthria splits the accuracy of S-word estimates
Twenty of the 42 participants scored below 4 on the ALS Functional Rating Scale Revised, indicating dysarthric speech; the other 22 had typical speech. Estimation errors were similar for the two groups in the plotted results, which the authors read as strong generalisation. But a substantial group disparity emerged in NRMSE: 0.060 for typical speakers versus 0.117 for dysarthric speakers, which the authors attribute to higher speech recognition word error rates and segmentation inaccuracies disproportionately affecting dysarthric speech and reducing accuracy for S-words.
The authors also note that some VFIs, particularly for dysarthric speakers, exceeded 12, producing more errors in the top-right region of the scatter plots. This is the population the automation is meant to help, since speech and motor deficits are exactly what confound traditional performance metrics. The paper's proposed response is future work incorporating supplementary tasks such as reading paragraphs and picture descriptions, particularly for dysarthric populations with elevated word error rates.
What a 42-participant study can and cannot support
The cohort is 42 people with MND recruited through CognoMND, an adaptation of the CognoSpeak web platform: mean age 64.7 years, 26.2% female, mean ECAS total 110.8, mean VFI 4.2 for P-words and 4.8 for S-words. All performance figures come from cross-validation within this single dataset, so they describe how well the models fit held-out folds of these participants, not how the pipeline would perform in a new clinic or on people with other conditions. The authors' own explanation for classical models beating deep learning is the small dataset, where simpler models generalise better.
The clinical value claimed is workflow rather than diagnosis. The pipeline produces clinician-interpretable outputs, including lists of correct words and errors, that can be shared with health care professionals, and the authors argue it reduces bias and subjectivity in identifying correct words and timing. Those are plausible benefits of automation, but the paper measures estimation accuracy against VFI values, not whether clinicians using the output make better or faster cognitive assessments.
Where automated VFI estimation earns its place
The paper's most decision-relevant comparison is not automatic versus manual scoring but feature choice: clinically interpretable timing and word-level measures consistently outperformed both traditional acoustic features and self-supervised embeddings, and the automatic clinical features matched manual ones closely enough that the difference was not statistically significant. For anyone building similar speech-based cognitive tools, that argues for prioritising precise alignment and pause structure over larger pretrained representations, at least at this sample size.
The second priority is the scenario split the results themselves create. For typical speech, the pipeline estimates VFI well on both letters; for dysarthric speakers, S-word accuracy degrades, with NRMSE 0.117 versus 0.060, and the authors point to supplementary tasks as the next step. The practical reading is that automated VFI estimation is most defensible today as a P-word, generation-phase measure in research settings, with dysarthric S-word scoring treated as the case that still needs either better recognition or additional tasks before it can carry clinical weight.
Sources used in this answer
Automatic estimation of verbal fluency index in people with Motor Neuron Disease using ASR alignment and pause modelling
A WhisperX and Silero pipeline with clinically interpretable timing features estimated the ECAS Verbal Fluency Index in 42 people with motor neuron disease at R² 0.856 and NRMSE 0.053 for P-words and R² 0.832 and NRMSE 0.083 for S-words, outperforming acoustic and self-supervised embedding baselines, with lower S-word accuracy for dysarthric speakers.