How robust are speech-driven financial voice assistants to regional British accents?

Accent errors reach the tool call, not just the transcript

A speech-driven financial assistant is a cascade: streaming ASR turns audio into text, an LLM reads that text to infer intent and emit a tool call, and text-to-speech returns a spoken answer. The ASR transcript is the only textual representation of customer intent the LLM sees, so an error there is not merely a transcription blemish.

The paper argues the cost of those errors is unequal. Word error rate weights every word equally, but a substitution in a merchant name, monetary amount, date, duration, or temporal expression can silently redirect a tool call, while an error in a less informative word may leave task completion intact. That is why the study measures tool-calling Exact Match alongside WER rather than treating transcription accuracy as the endpoint.

CavaBench pairs four accent groups with financial queries

CavaBench is described as the first internally collected, human-labeled audio benchmark for spoken financial queries, balanced across four British-accent categories. It was recorded by 90 participants (50 male, 40 female) through an in-house interface in a clean acoustic environment, deliberately isolating accent as the primary variable.

Each speaker recorded at least one round of 20 queries drawn from a pool of 200 realistic financial queries covering spending summaries, budget tracking, merchant lookups, trend analysis, and numerical queries. The result was 3,079 utterances, of which 2,919 were retained after quality inspection, with reference transcripts taken directly from the prompt text. Utterance duration averaged 4.73 seconds (range 2.16–20.82 seconds; 3.94 hours total).

Self-reported accent, collected as 76 raw free-text values, was grouped into Scottish (about 685 utterances, about 23 speakers), Southern English (about 710, about 23), Other British (about 747, about 20, bundling Northern Irish, Mancunian, Yorkshire, Welsh, and Midlands), and Non-British (about 784, about 26). The grouping supports balanced group-level comparison, but it also means the Other British category mixes several distinct regional accents into one label.

Fine-tuning narrows the accent spread more than the average

In the clean condition, fine-tuning helped the smaller streaming model without widening the accent gap. Nemotron Streaming improved overall WER from 18.2% to 16.1% and narrowed its max-min accent spread from 2.3% to 1.5%, the paper's fairness metric. Applying the same procedure to Parakeet narrowed the accent spread from 2.7% to 1.7% while overall WER moved only marginally, from 16.6% to 16.5%, and Exact Match did not improve.

Accent sensitivity was model-dependent. Qwen3-ASR-1.7B Chunking had the largest spread of any model at 9.4%, driven by a collapse on Scottish speech at 22.2% WER. Cohere Offline reached the third-lowest overall WER at 15.3% with one of the tightest spreads at 1.6%. In the clean-only condition, overall Exact Match ranged from 75.1% (Nemotron Base-LA6) to 80.7% (Qwen3-ASR-1.7B Offline), and fine-tuning raised Nemotron Streaming's Exact Match from 75.5% to 78.2%.

The distinction matters because a lower average WER and a narrower accent spread are different achievements. Parakeet's fine-tuning is the clearest case: the spread improved while the average barely moved and task-level accuracy did not follow.

Acoustic conditions degrade models unevenly

Robustness fell monotonically across the simulated conditions: Clean at 6.0% WER, then Reverb at 9.8%, Noise at 11.8%, Telephony at 12.9%, Overlap at 20.0%, and Environment at 28.6%. The Environment condition, which combines degradations, was substantially harder than any single one.

Models did not degrade at the same rate. Whisper Turbo, which differs from its distilled variant only in decoder size, showed the steepest single-condition collapse at 29% WER under Environment. Qwen3-ASR-1.7B Offline was the most robust throughout, retaining the highest Exact Match except under Reverb, with Cohere Offline behind only the two Qwen3-ASR-1.7B variants. The Nemotron Streaming baseline, comparable in scale to Parakeet but two to four times smaller than the Whisper and Qwen3-ASR-1.7B models, showed less robustness, which the authors attribute to less acoustically diverse pretraining data and model size. Fine-tuned models reduced the gap at Telephony, matching that corpus's own domain.

WER predicts tool-calling accuracy except for Scottish

Averaged over acoustic conditions, overall WER and Exact Match were strongly anti-correlated across models, at r = -0.93. That supports using WER as a general signal for model selection. The relationship was not stable across accent groups, however: correlations ranged from -0.87 to -0.98 for three of the four groups but weakened substantially for Scottish, at r = -0.29, largely because of Qwen3-ASR-1.7B Chunking's disproportionate Scottish WER spike.

All four Nemotron Streaming variants showed strong within-model correlations, from -0.74 to -0.96, and the same ranking pattern: Southern English achieved marginally higher Exact Match than Scottish despite slightly higher WER. The authors read this as evidence that the pattern holds across fine-tuning and lookahead settings.

The paper's interpretation is that different accents expose different challenges. A single model's WER spike breaks the correlation for Scottish, while for Other British the difficulty appears to come from accent diversity itself rather than from any one accent. Aggregate WER also masks the unequal cost of errors, since preserving task-critical tokens matters more than reducing total word errors.

Lookahead buys modest gains at higher latency

Comparing LA6, the shorter-context and lower-latency setting, against LA13, the longer-context and higher-latency setting, isolates the lookahead effect. The baseline configuration improved from 20.0% WER and 64.6% Exact Match at LA6 to 18.2% and 66.5% at LA13. The fine-tuned configuration improved from 17.6% and 66.4% to 16.1% and 67.9%.

That corresponds to a consistent but modest improvement of 1.5–1.8 percentage points in WER and 1.5–1.9 points in Exact Match. The paper frames accent robustness and streaming efficiency as competing requirements that must be optimized jointly, because the accent gap widens or narrows with the chosen latency operating point rather than staying fixed.

The robustness gap traces to training data, not architecture

The authors attribute the robustness gap to training data rather than architecture. Their supporting observation is that targeted fine-tuning closed much of the gap for the smaller streaming model, which points to efficient streaming models trained on more diverse data as a path toward matching offline counterparts.

This is the paper's own explanation, and it sits alongside a measurement boundary worth keeping in view. CavaBench was recorded in a clean acoustic environment with accent isolated as the primary variable, and acoustic robustness was evaluated separately under simulated conditions. The benchmark is also internally collected, and its Other British category aggregates several distinct regional accents, so group-level results for that category reflect bundled diversity rather than a single accent.

The practical priority the evidence supports is not simply picking the model with the lowest average WER. It is checking whether a candidate's accent spread and task-level Exact Match hold up for the specific accent groups and acoustic conditions a deployment will actually encounter, since a model can look adequate on aggregate WER while failing disproportionately on one group.

Sources used in this answer