How does VOICE-AE automate CTCAE scoring from ambient clinical audio?

From recorded visit to graded toxicity

VOICE-AE is a multi-stage pipeline, not a single model. Audio from clinician-patient encounters is transcribed with Whisper Large-v3, then segmented and labeled by speaker with WhisperX and the pyannote diarization model. A first LLM stage, Gemini 2.0 Flash, assigns speaker roles such as physician or patient and pulls out question-answer pairs that mention symptoms or toxicity. A second stage, Gemini 2.5 Pro, maps each extracted mention onto one of 10 predefined genitourinary and gastrointestinal CTCAE v5.0 items and assigns a severity grade from 0 to 5. When the same toxicity is graded from several transcript segments in one encounter, the highest grade is kept.

The design choice that matters for interpretation is the constrained output space: the model does not grade arbitrary toxicities, only the 10-item list it is given. Speech recognition and diarization ran locally on institutional hardware, while the LLM stages ran in a HIPAA-compliant cloud environment, so no audio left the institution but transcripts did reach the models.

The reference standard is expert transcript reading

Six blinded raters graded all 10 toxicities for every encounter using only the diarized transcript and the toxicity list; they could request the original audio but did so in just 4 of 234 transcript reviews (1.7%). Hierarchical clustering on pairwise weighted kappa split the panel into an expert cluster (two radiation oncologists and an advanced practice provider) and a trainee cluster (two residents and a research fellow). The expert cluster became the adjudication panel, and its consensus grades formed the reference standard.

This matters because the study measures agreement on how a transcript should be interpreted, not accuracy against what the patient actually experienced. The authors state that no external validation against chart-documented toxicity or patient-reported outcomes was performed, and that a reference standard built from clinicians reading transcripts inherits the same under-reporting limitation that motivated the work. The panel was also selected post hoc from the same ratings later used to build the reference standard, a procedure the authors note is expected to favor the panel's later-reported agreement.

High raw accuracy rests on a grade-0 majority

Across 31 patients and 39 encounters, the study produced 390 encounter-toxicity assessments, and 300 of them (76.9%) carried a reference grade of 0. Against the adjudicated standard, VOICE-AE reached a mean weighted kappa of 0.717, ordinal Krippendorff alpha of 0.669, exact grade match of 82.6% and within-one-grade accuracy of 99.2%. Mean absolute error was 0.18 and RMSE 0.44.

The raw percentages need the baseline beside them: a classifier that always predicted grade 0 would score 76.9% exact match and 93.6% within one grade on this distribution. VOICE-AE therefore beats that trivial baseline by 5.6 percentage points on both measures, and the chance-corrected statistics are the more informative summary. Restricting analysis to the 32 encounters with at least one non-zero reference grade left ordinal agreement essentially unchanged (alpha 0.694 versus 0.669), which argues the agreement is not purely zero-to-zero matching.

Grade 1 and grade 2 are where the errors sit

Per-grade performance shows the weak point. Grade 0 reached an F1 of 0.91, but grade 1 reached 0.58 and grade 2 reached 0.57; the weighted-average F1 of 0.83 is dominated by the grade-0 majority. For binary presence versus absence of any toxicity, micro-averaged sensitivity was 77.8%, specificity 89.0%, precision 68.0% and negative predictive value 93.0%. In practical terms, about one in five true toxicities was missed and about one in three positive calls was a false positive.

The confusion matrix shows the same pattern from another angle. Of 90 assessments with a non-zero reference grade, 55 (61.1%) were graded exactly, 26 (28.9%) were under-graded and 9 (10.0%) were over-graded, so under-grading outnumbered over-grading nearly three to one. Half of grade 2 events were misclassified, with 8 of 24 (33%) assigned a lower grade. The cohort contained a single grade 3 event and no grade 4 or 5 events, so performance at the severity levels that adverse-event reporting mainly exists to capture could not be evaluated.

Some toxicities are solved and others are not

Performance varied widely by toxicity. Flatulence and urinary incontinence matched the reference perfectly (Krippendorff alpha 1.0, F1 1.0), dysuria was near-perfect (alpha 0.91, F1 0.92), and urinary frequency and diarrhea were strong. At the other end, urinary tract obstruction (alpha 0.08, F1 0.17), urinary tract pain (alpha 0.31, F1 0.36) and constipation (alpha 0.47, F1 0.53) were poorly captured.

The comparison with human agreement helps separate two causes. Urinary tract obstruction and urinary tract pain also had the lowest human inter-rater agreement (alpha 0.11 and 0.41), suggesting difficulty intrinsic to the CTCAE definitions. Constipation, by contrast, had much higher human agreement (alpha 0.80), so its poor AI performance looks more like a specific extraction problem than inherent ambiguity. The authors also note that Cohen's kappa is attenuated at low prevalence and report PABAK values that were substantially higher for several low-prevalence toxicities.

The model sits just below expert-to-expert agreement

Adding Gemini 2.5 Pro as a seventh rater placed it inside the expert cluster, between the advanced practice provider and the radiation oncologists, with its highest pairwise agreement against the junior radiation oncologist (kappa 0.687) and the senior radiation oncologist (0.654). That placement alone could be read as expert-level performance, but the like-for-like comparison is the fairer test.

Restricted to the three expert-cluster raters, mean pairwise weighted kappa among the experts was 0.681 (range 0.652-0.709), while mean weighted kappa between VOICE-AE and those same three raters was 0.666 (range 0.654-0.687). For presence versus absence the gap widened: 0.682 among experts versus 0.580 for the model. The whole-panel comparison that appears to favor the model (0.599 versus 0.586) only does so because the 15 human-human pairs include lower-agreement trainee pairs. Run-to-run stability was high, with 99.2% of assessments reproduced within one grade and inter-run alpha of 0.920, so variability between runs was smaller than the pipeline's disagreement with the reference standard.

Feasibility for documentation, not autonomous grading

The paper's own claim is feasibility: a fully automated path from ambient clinical audio to structured, graded CTCAE output, with agreement falling within the range observed among expert human raters. The authors explicitly state that accuracy at grade 2 or above and detection of severe toxicity remain unproven. The scope is narrow in ways that bear on any use: 31 patients at a single institution, prostate cancer radiotherapy, only genitourinary and gastrointestinal toxicities, and recordings made in a controlled research setting rather than routine clinic noise.

For a reader deciding what to take from this, the useful split is by grade and by toxicity rather than by the headline kappa. The pipeline's strength is ruling out or confirming the absence of a toxicity (negative predictive value 93.0%) and staying within one grade; its weakness is the grade 1-2 boundary and specific items such as urinary tract obstruction and pain, where even human raters disagreed. The matched comparison to expert raters, not the whole-panel one, is the number to carry forward: 0.666 against 0.681 among experts.

Sources used in this answer