Can multimodal AI grade medical student OSCEs in routine deployment?

A first-pass scorer with humans on the low tail

The paper reports what its authors describe as the first prospective, single-center deployment of an integrated multimodal AI system for grading undergraduate medical OSCEs. The system, MAPLES (Multimodal Assessment Pipeline for Learning Encounter Scoring), is a rubric-driven zero-shot multimodal large language model pipeline: it scores each rubric item directly from the assessment criteria without task-specific model training.

The workflow is the part that matters for interpreting every number that follows. Each encounter produced three artifact streams: a written clinical note, an audio recording, and synchronized three-camera video. MAPLES scored the full item set for all learners. Learners whose aggregate scores fell in the bottom 5% for notes or the bottom 10% for audio/video were then routed to standardized patient evaluators (SPEs), who rescored those items blinded to the AI scores. Disagreements beyond a predefined tolerance were escalated to physicians, who saw both scores and their source evidence and set the final operational score.

So the AI did not grade alone and humans did not grade everything. The AI produced the routine first pass; humans concentrated on the learners most likely to need remediation and on the items where two graders diverged.

Agreement figures describe the routed tail only

In the routed review set, exact AI–SPE agreement was 72.0% (95% CI 69.1–74.5) and tolerant agreement was 87.7% (95% CI 86.6–89.0). These two metrics answer different questions and should not be swapped. Exact agreement required identical scores. Tolerant agreement required identical scores on binary items but allowed a one-category difference on items with three or more ordinal levels.

The distinction matters because most disagreements were small. AI assigned higher scores to 1,060 reviewed items (21.3%) and lower scores to 330 (6.6%), and the paper reports that most AI–SPE disagreements were limited to a one-level response-category difference. That pattern is what justifies a tolerance window before escalation rather than treating every mismatch as an error.

The scope condition is explicit in the paper: because review was intentionally concentrated on AI-flagged low scorers rather than a representative sample, these agreement and adjudication results describe performance within the routed lower-scoring tail and should not be read as estimates of AI–human parity across the full cohort.

Notes agree most exactly; video agrees most tolerantly

Agreement varied by evidence modality. Exact agreement was 80.0% for note items, 61.6% for audio, and 69.4% for video. Tolerant agreement reversed the ranking: 85.7% for notes, 89.2% for audio, and 92.4% for video.

The paper's reading is that audiovisual disagreements were mostly one-category differences on multi-level judgment scales rather than gross errors. That interpretation is consistent with the direction data: AI scored higher than the SPE more often for notes (16.2% versus 3.8%) and audio (29.5% versus 8.9%), while video disagreements were more balanced (16.3% higher versus 14.3% lower).

Response-scale structure also mattered. Item scale-stratified quadratic-weighted kappa was 0.60–0.75 for two- to five-level items but 0.46 for six-level items, and exact agreement decayed from 82.3% at two levels to 36.5% at six levels while tolerant agreement held until the six-level extreme. Finer-grained scales are where the AI–SPE match weakens most.

Physicians sided with AI in most escalated disagreements

Of 753 escalated items, excluding disagreements attributable to rubric and data input errors left 616 genuine disagreements as the adjudication set. Among these, physicians matched the AI score on 468 items (76.0%; 95% CI 72.0–79.8), the SPE score on 117 items (19.0%), and neither on 31 items (5.0%).

That preference held in every modality but weakened from text to audiovisual evidence: 81.6% for note items (n = 376), 69.6% for audio (n = 207), and 51.5% for video (n = 33). Scores matching neither grader were rare for notes (0.8%) but more common for audio (11.6%) and video (12.1%). Physicians also endorsed the AI score both when the AI had scored higher than the SPE (76.9%) and when it had scored lower (69.6%), so the pattern was not confined to cases where the AI awarded more credit.

The paper is careful about what this does and does not show. Adjudicators saw both scores with their sources identified, so anchoring or automation bias may have influenced the matching proportions. The authors state that these proportions describe physician score matching within a selected, source-visible adjudication set and should not be interpreted as comparative grader accuracy or superiority.

AI errors were over-permissiveness and misplaced documentation

Physician-commented adjudications were classified by contributing factor. Among 127 AI-attributed instances, the largest categories were over-permissive scoring (48, 37.8%) and wrong documentation placement (32, 25.2%), followed by rubric lack of specificity (16, 12.6%), evidence oversight (12, 9.4%), and fabricated evidence (12, 9.4%).

Among 131 SPE-attributed instances, the leading factors were different: evidence oversight (46, 35.1%) and clinical knowledge (32, 24.4%), with over-permissive scoring at 24 (18.3%). The paper notes that SPE graders failed to recognize equivalence between rubric terminology and what was written or said in the encounter, a category where AI error was minimal (under 2%).

The authors argue that AI errors are more amenable to prompt and instruction engineering than human errors, and that improving rubric specificity and clarity was, in expert opinion, the single largest improvement available. Fabricated evidence accounted for a small share of AI error (under 10%), pointing to hallucination risk that remains relevant in multimodal assessment.

Effort falls 92.3% while review work remains

The AI-assisted workflow required 5,616 human scoring passes: 4,973 independent SPE rescoring passes and 643 physician adjudication passes. A counterfactual single-pass manual workflow would have required one pass for each of the 72,907 retained item-level scores, which the paper reports as a 92.3% reduction. Reductions were 94.0% for note, 88.3% for audio, and 88.9% for video items.

The paper frames this as redistribution rather than removal of human judgment. SPEs retained routed review of the low tail and physicians retained adjudication, and the same review process surfaced ambiguous, defective, and miswritten rubric items that had previously gone uncaught. Faculty reported shifting attention from how to grade toward what skills to evaluate.

Two operational figures accompany this. AI token costs were estimated at approximately $1.73 per encounter, with a projected cohort cost of about $1,900 across 1,110 nominal encounters, and total turnaround from examination to grade reports fell from several weeks or months to under two weeks. These are planning estimates based on stated pricing assumptions, not measured invoices.

This is a deployment study, not a grader comparison

The paper states directly that its contribution is best read as an assessment-system deployment, not a comparison of graders. Physician alignment with AI within this workflow indicates that escalated disagreements rarely warranted departing from the AI score, but it does not establish that AI grades better than SPEs. That question would require blinded adjudication, uniform sampling across the score distribution, and a high-quality independent expert reference grade.

Several boundaries follow from the design. The deployment was prospective but single-center, at one simulation center with 222 second-year students, ten station forms, and rubrics written by local faculty. Learner-level concordance analyses included only learners selected by low AI scores, so they characterize ordering within the routed review set rather than the full cohort. Item-level OSCE data are FERPA-protected educational records and are not publicly available.

The routing thresholds themselves encode a judgment: the 5% note and 10% audio/video windows were chosen partly because these OSCEs are primarily formative and because historical pass/fail shifts occurred only within roughly the bottom 3% of learners. A high-stakes summative use would need a different review policy and its own validation.

The review threshold, not the model, carries the risk

The most consequential number in this paper is not the 76.0% physician match rate. It is the routing threshold that decided which 28 learners out of 222 received independent human review: 12 through the note pathway, 23 through audio/video, with seven selected by both. Everything downstream — the 4,973 SPE rescoring passes, the 616 adjudicated disagreements, the 92.3% effort reduction — is a property of that selection rule as much as of the model.

That makes the threshold the place where safety and savings trade off. Widening the review window buys more human oversight of borderline learners and gives back some of the effort reduction; narrowing it saves more passes but leaves more AI-only scores in the record. The paper's own justification for the wider 5–10% window was an added margin of safety beyond the roughly bottom 3% where historical pass/fail decisions moved.

For a program considering this workflow, the transferable lesson is to treat the review threshold as the primary design parameter and to validate it against local grading distributions, rather than to import the reported agreement percentages as a general benchmark. The agreement figures here describe a routed low-scoring tail at one institution; the threshold is what determines whether that tail is the right place to spend human attention.

Sources used in this answer