Can Large Audio Language Models Detect Dysarthric Speech With Acoustic Features?

Zero-shot models sit at chance level

The paper's starting point is a negative result that shapes everything after it. Asked to label recordings as healthy or dysarthric without any task-specific training, Phi-4-multimodal scored 50.0±0.0% and Qwen2-Audio scored 51.0±0.0% speaker-level accuracy on PC-GITA. On a two-class problem, that is coin-flip performance. The authors read this as evidence that these models do not inherently possess the capability to perform dysarthric speech detection, even though they handle many other audio tasks well.

This matters for how the rest of the results should be read. Any gain reported later comes from adapting the model to the task, not from a latent diagnostic ability that a prompt merely surfaces. The paper's own framing is that fine-tuning accounts for the majority of the improvement over zero-shot performance.

Semantic encoders skip the acoustic detail

The motivation for adding features is a claimed gap in how these models listen. The paper cites prior work finding that large audio language models rely primarily on semantic speech encoders such as the Whisper encoder, and therefore largely neglect low-level acoustic features. Qwen2-Audio's audio encoder is indeed based on Whisper-large-v3, which the paper describes alongside its 128-channel mel-spectrogram front end.

That gap matters because dysarthria is described in the clinical literature as manifesting in characteristics such as monopitch, monoloudness, breathiness, and irregular speech rate. These are properties of the acoustic signal rather than of the words spoken. If the encoder mostly captures semantic content, the cues a clinician would listen for may not reach the language model. The paper's proposed fix is to compute a small set of descriptors separately and write their numeric values into the text prompt.

Six numbers written into the prompt

The feature-augmented framework keeps the audio input and adds a sentence of text. Using the OpenSMILE framework, the authors compute the mean and standard deviation of pitch in Hz, the mean and standard deviation of perceptual loudness in dB, the mean harmonic-to-noise ratio in dB, and the mean speech rate in syllables per second. The prompt also states the speaker's gender and language. Raw numeric values go directly into the prompt rather than being converted into categorical labels.

The authors state that these descriptors were chosen because they are commonly associated with dysarthric speech in the clinical literature, not through any feature-selection or optimization procedure. They describe the set as an illustrative starting point that the framework could extend. The comparison condition is a standard prompt that supplies only the audio and the task description, with no demographics and no feature values.

Fine-tuning does most of the work

The ablation separates two changes that could otherwise be conflated. Moving from zero-shot to standard fine-tuning raised Phi-4-multimodal from 50.0±0.0% to 79.0±5.1% and Qwen2-Audio from 51.0±0.0% to 84.0±2.5%. Adding the acoustic features and demographics on top of fine-tuning raised them further, to 83.0±2.2% and 85.0±0.8% respectively.

So the ordering is clear: adaptation to the task produces the large jump, and feature augmentation adds a smaller but consistent gain. The paper also notes that feature augmentation reduced the standard deviation across seeds, which it interprets as the models becoming more robust to how the cross-validation folds are constructed. That robustness claim rests on the spread across three random seeds, not on a separate experiment.

Both models clear the deep learning baselines

The headline comparison is against two self-supervised baselines built the way the paper describes current state of the art: representations from wav2vec 2.0 or XLS-R, averaged over time, passed through two linear layers to a two-class output. On PC-GITA with 10-fold cross-validation, the wav2vec 2.0 baseline reached 82.3±3.1% and the XLS-R baseline 80.3±1.7% speaker-level accuracy.

The feature-augmented models reached 83.0±2.2% for Phi-4-multimodal and 85.0±0.8% for Qwen2-Audio. Qwen2-Audio is the strongest configuration in the paper. The margins are modest relative to the baseline standard deviations, and the paper's claim is stated at the level of outperforming the baselines rather than of a large separation. Note also that the standard fine-tuning condition for Qwen2-Audio, at 84.0±2.5%, already sits above both baselines without any added features.

Harmonic-to-noise ratio ranks first

A second ablation, run on Phi-4-multimodal as a representative model, ranks the four descriptors by fine-tuning with each one individually. The order from most to least effective is harmonic-to-noise ratio, speech rate, loudness, and pitch. The authors then add features incrementally in that order and report that performance improves overall as features accumulate, with all four combined giving the best result.

This ranking is specific to one model and to the PC-GITA setting, and the paper presents it as a ranking rather than as a claim that harmonic-to-noise ratio carries the diagnostic signal on its own. The task description was held general and not modified as features were introduced, so the comparison isolates the feature values rather than changes in wording.

Accuracy is decided per speaker, not per clip

The reported figures are speaker-level accuracies, which is a different quantity from clip-level classification. Recordings are segmented into 30-second chunks with 50% overlap, with the final residual segment of each utterance retained, and the model's diagnosis for a speaker is obtained by voting among all utterances belonging to that speaker. A reader comparing these numbers to clip-level results elsewhere would be comparing different units.

The dataset is PC-GITA, a Spanish corpus with recordings from 50 patients with Parkinson's disease and 50 neurotypical speakers, balanced at 25 male and 25 female in each group. The experiments use the monologue, read-text, and sentence subsets, resampled to 16 kHz. Evaluation uses the standard 10-fold cross-validation framework, repeated with three random seeds, and the reported values are means with standard deviations across those runs.

One corpus, one condition, two models

The evidence supports a narrow but useful conclusion: on PC-GITA, with Parkinson's disease as the source of dysarthria, fine-tuning two specific audio language models with a prompt containing six acoustic descriptors and speaker demographics produced speaker-level accuracy above two self-supervised baselines. The paper does not test other corpora, other languages, or other neurological causes of dysarthria, and it does not report a comparison against a clinician's assessment.

Within that boundary, the practical priority the paper establishes is the ordering of interventions rather than the feature list itself. Fine-tuning is what moves a model off chance level; the acoustic descriptors and demographics are what close the remaining gap to and past the baselines, and they also tighten the spread across seeds. For Qwen2-Audio the feature gain over standard fine-tuning is 1.0 point, while for Phi-4-multimodal it is 4.0 points, so the value of the augmentation is model-dependent and largest where the base fine-tuned model is weakest.

Sources used in this answer