Can large language models reliably score open-ended responses in a diabetes precision medicine course?

Three outcomes, one course, 435 completers

The paper reports an observational single-group pre-post evaluation of an online diabetes precision medicine course, delivered through the Stanford Data Ocean platform and supplemented by a structured two-week synchronous online workshop. Of 731 learners who enrolled and attended the workshop, 435 completed the self-paced course and all required evaluation components, a completion rate of 59.5%; the remaining 296 learners had not completed all required components by the extraction date.

Three outcomes were tracked before and after the course: multiple-choice knowledge, self-reported confidence on a 1-5 scale, and rubric-scored open-ended responses. The open-ended part matters because selected-response formats capture factual recall well but not a learner's ability to explain and integrate precision medicine concepts in their own words. The study collected 4,350 open-ended responses across five instructional domains, which is the volume that made manual scoring impractical and motivated the LLM workflow.

Because the design has no control group, the pre-post changes are associations with course completion, not demonstrated effects of the course. The authors list repeated testing, regression to the mean, external learning resources, the synchronous workshop, possible AI assistance, and differential motivation as alternative explanations.

How three models reach a consensus score

The workflow is a pipeline rather than a single prompt. It first screens each response for completeness, then generates a question-specific rubric, then has three LLMs score each response independently on a three-level ordinal scale (1 = Beginner, 2 = Intermediate, 3 = Advanced). A consensus rule assigns the final score, and responses with low agreement among the three models are flagged for human review.

Completeness screening classified 1,131 of 4,350 expected responses (26.0%) as insufficient and 3,219 (74.0%) as scorable. Insufficient responses were kept in the dataset with a final score of 1 and labeled Beginner. Among the 3,219 scorable responses, 2,146 (66.7%) received identical scores from all three models, 1,061 (33.0%) received two distinct scores, and 12 (0.4%) received three distinct scores. Across all 4,350 expected responses, those 12 low-agreement cases represent 0.3%, the only responses that met the predefined criteria for human review.

The final LLM-assisted scores classified 1,780 responses (40.9%) as Beginner, 1,917 (44.1%) as Intermediate, and 653 (15.0%) as Advanced. The logged workflow consumed 2,291,814 tokens and 6,228.3 seconds (103.8 minutes) of aggregate API-call latency; because some calls ran concurrently, that figure is not elapsed wall-clock time.

Exact agreement is moderate, adjacent agreement is high

Validation used a stratified random sample of 300 responses, scored by human raters as the reference. The final consensus score matched the final human score exactly for 75.3% of responses, fell within one rubric category for 99.3%, and produced major disagreements, defined as a two-category gap, for 0.7% (two responses). The quadratic weighted kappa was 0.78 and the mean absolute difference was 0.25.

Individual models were close behind: exact agreement with the final human score ranged from 71.0% to 74.3%, adjacent agreement from 99.0% to 99.7%, quadratic weighted kappa from 0.74 to 0.78, and mean absolute differences from 0.26 to 0.30. The consensus step therefore bought a modest gain over any single model rather than a dramatic one.

A useful comparison sits inside the same sample: the three LLMs agreed unanimously on 74.0% of responses, while human raters were unanimous on 65.7%. Model-to-model agreement was thus higher than human-to-human agreement, which is why the authors treat agreement with humans as reliability evidence rather than proof that the rubric captures conceptual understanding. In the 12 low-agreement cases, human raters were unanimous for only 3 of 12 (25.0%), and the final consensus matched the final human score exactly in 75.0% of them with 100% adjacent agreement.

Knowledge rose 30 points, confidence 1.32 points

Knowledge increased by 30.0 percentage points (95% CI 28.5-31.6; Cohen's dz = 1.81; p < 0.001). At the learner level, 387 participants (89.0%) improved, 48 (11.0%) were unchanged, and none scored lower after the course.

Overall confidence rose from 2.74 ± 1.21 to 4.06 ± 0.64 on the 1-5 scale, a mean difference of 1.32 points (95% CI 1.22-1.43; dz = 1.19; p < 0.001). Domain-specific confidence gains ranged from 1.20 to 1.48 points. At the learner level, 351 participants (80.7%) had higher average confidence, 48 (11.0%) were unchanged, and 36 (8.3%) had lower average confidence.

Open-ended response scores rose by 0.39 points on the three-level rubric (all three outcomes p < 0.001). These three numbers are not interchangeable: one is a percentage-point change on a knowledge test, one is a shift on a self-report scale, and one is a movement on an ordinal rubric. Confidence in particular is self-reported and may reflect response bias rather than objective competence.

The three outcomes barely align at the learner level

The study also asked whether the same learners improved on all three outcomes. They largely did not. Pairwise agreement between change categories was low despite cohort-level improvement: quadratic weighted kappa was 0.07 for knowledge versus confidence, -0.05 for knowledge versus open-ended responses, and -0.01 for confidence versus open-ended responses.

Spearman correlations between continuous change scores were similarly weak. Knowledge and confidence changes showed a small positive correlation (rho = 0.13, p = 0.005), while knowledge and open-ended response changes (rho = 0.06, p = 0.196) and confidence and open-ended response changes (rho = 0.08, p = 0.099) were not significantly correlated.

The authors read this as evidence that multiple-choice knowledge, self-reported confidence, and written conceptual understanding capture related but non-interchangeable dimensions of learning. For readers, the practical consequence is that a cohort-level gain on one measure says little about which individual learners gained on another, so the three outcomes should be reported and interpreted separately rather than collapsed into a single learning score.

What the validation sample and rubric cannot settle

The agreement figures come from 300 responses drawn from one course, scored against human raters using a three-level ordinal rubric. That rubric supported scalability and interpretability but may compress meaningful differences within the Beginner, Intermediate, and Advanced categories. Agreement with humans supports reliability of score assignment; it does not by itself establish that the rubric captures the broader construct of conceptual understanding, which the authors say would require known-group validity and associations with external measures.

The workflow used three widely available hosted models, including a model alias and a preview model, so results may not generalize to other models or future versions. Temperature zero does not guarantee deterministic inference, and exact outputs may vary across executions or after provider updates. The study did not evaluate locally hosted open-weight models, and free-text responses could contain incidental identifying information even after account names were replaced with study-specific IDs.

Open-ended responses also do not solve assessment integrity. Learners may use AI tools to generate written answers, and the authors note that students may attempt to manipulate LLM graders when prompts and scoring systems are exposed. The rationale for open-ended items here is broader construct coverage, not integrity enforcement. Learners who completed all assessments also differed from incomplete learners on some baseline characteristics, including gender, degree, and study area, which may introduce selection bias.

Treat the 0.3% review rate as the operational claim

The paper's most transferable claim is not that LLMs grade as well as humans. It is that a specific quality-controlled pipeline, combining question-specific rubrics, independent scoring by three models, predefined consensus rules, low-agreement flagging, and human validation, reduced the human review burden to 12 of 4,350 responses (0.3%) while keeping major disagreements at 0.7% in the validation sample. The efficiency came from the workflow design, not from using an LLM alone.

That framing sets the decision threshold for anyone considering a similar setup. The relevant questions are whether adjacent agreement near 99% is sufficient for the intended use, whether a three-level rubric preserves the distinctions the course cares about, and whether the 0.3% flagging rate holds when models, prompts, or rubrics change. Because the authors position the work as establishing operational feasibility and preliminary validity for research and formative assessment, the appropriate use is formative feedback with targeted human adjudication, not high-stakes grading without human oversight.

The learning-outcome side of the paper should be read on its own terms: a single-group pre-post design in 435 completers showing gains in knowledge, confidence, and open-ended scores, with weak learner-level alignment across the three. The scoring workflow and the course-outcome findings are separate contributions, and the stronger evidence belongs to the workflow.

Sources used in this answer