Can LLMs reconstruct longitudinal clinical trajectories in chronic liver disease?

A pipeline that turns reports into variables

The paper asks whether open-source large language models (LLMs) can convert routine clinical free text into structured data at cohort scale, and whether the resulting records are good enough to rebuild a patient's disease history over time. Chronic liver disease surveillance generates long trails of radiology, pathology and transplant assessment reports, but because that information sits in prose it is hard to study at scale. The authors built a framework that poses a fixed list of questions to each report and records the answers as variables.

The evaluation ran in stages. A calibration set of 507 reports from 30 patients — 414 radiology, 65 pathology and 28 liver transplant assessment reports — was manually annotated to create ground truth. Four open-source models were compared against a regular expression baseline across 73 tasks: Llama-3.1-8B, Llama-3.3-70B, OpenBioLLM-70B and DeepSeek-R1-8B. The winning configuration was then applied to all 835 patients transplanted at the centre over the past decade.

Constrained decoding fixes format, not accuracy

A central technical choice was how to force models to return usable output. Unconstrained prompting simply asks the model for an answer; constrained decoding restricts generation to an allowed structure. Across 5,590 prompt-output pairs, constrained decoding reached 99.9% format adherence, against 87.4% for unconstrained prompting. With the Guidance package there were no invalid outputs; with Ollama, 4 of 5,614 outputs were invalid (0.07%). Unconstrained prompting failed on 708 of 5,614 outputs (12.61%), and the failure rate was higher for numeric questions (15.38%) than categorical ones (11.60%).

Reliability carried a cost. On a single NVIDIA A100-SXM4 80 GB GPU, unconstrained template prompting finished the calibration extraction in 73 minutes, while Guidance-based constrained decoding took 126.8 minutes, roughly 1.7 times longer. The Ollama implementation, run separately on an Apple M3 Pro chip, took 183.4 minutes. Format adherence and answer correctness are different measures: a perfectly formatted output can still contain the wrong value, which is why the paper benchmarks accuracy separately.

The larger general model beat the medical one

Llama-3.3-70B with Guidance was the best performer, exceeding 90% accuracy on 26 of 31 radiology questions and staying at or above 80% on the rest; across the full 73-task benchmark the abstract reports it above 90% on 59 tasks. The comparison with OpenBioLLM-70B, a medically fine-tuned model built on the earlier Llama-3.1 architecture, is the more surprising result: Llama-3.3-70B was significantly better on 5 of 31 questions (16.1%) and at least as good on the other 26, while OpenBioLLM never outperformed it.

Model size mattered less than expected. Llama-3.1-8B was outperformed by the 70B model on 8 of 31 questions (25.8%) but performed comparably on the remaining 23 (74.2%). Regular expressions were also competitive on categorical questions, matching or beating the top LLM on 13 of 31 (41.9%) and losing on 18 (58.1%), while LLMs were clearly stronger on numeric extraction. The authors suggest medical fine-tuning may help less here because medical benchmarks often use multiple-choice questions rather than extraction from real clinical text, and because OpenBioLLM is based on an older base model.

Cohort-scale extraction reproduced known risk factors

Applied to the full cohort, the pipeline analysed 22,493 reports and produced 37,125 datapoints covering 45 variables across 835 patients, with no manual annotation. The design used conditional questions so that, for example, a report with no liver lesions would not trigger prompts about lesion size. Patient-level summaries then collapsed the reports: binary variables such as cirrhosis were marked positive if any report indicated the feature, and ordinal variables such as lesion grade took the most severe instance across reports.

Analysis of the extracted data confirmed associations already established in the literature — male sex, viral hepatitis, smoking and diabetes with liver cancer development, alongside higher BMI and older age in the discussion. The authors also compared primary chronic liver disease diagnoses made at transplant assessment with diagnoses from pathological examination of the explanted liver, finding that most reclassifications shifted away from alcohol-related liver disease. These are associations within a transplant cohort, not causal claims, and the authors note the cohort is cancer-enriched and shaped by transplant selection criteria.

Event-level timelines expose the real weakness

Reconstructing a timeline is harder than filling a table. For lesion timelines, each timepoint is a scan date relative to transplant, annotated with the number of lesions of each LI-RADS grade. For treatment timelines, dates of resection, ablation and embolisation are converted to day offsets from transplant. Here the models were scored on individual events rather than across all reports, and the picture changed.

For embolisation identified from radiology reports, Llama-3.3-70B reached sensitivity 0.77 versus 0.67 for Llama-3.1-8B, with both at 0.77 precision. Both found all ablation events with one hallucinated instance (sensitivity 1.00, precision 0.86). Resection was the failure point: Llama-3.3-70B added eight resection instances across eight patients, dropping precision to 0.20 despite sensitivity of 1.00, while Llama-3.1-8B added two and reached precision 0.50. Both models did better on the more standardised liver transplant assessment reports, where Llama-3.3-70B achieved perfect precision and sensitivity for embolisation and ablation and added only one spurious resection event (precision 0.67).

The authors explain the gap between aggregate and event-level scores directly: aggregate metrics include many true negatives, such as lesion-free scans, whereas timeline evaluation counts errors only against true-positive events. That is why high benchmark accuracy and unreliable timelines can coexist, and why the paper argues for task-specific, event-level evaluation before clinical use.

Errors cluster around time, syntax and inference

The qualitative error review groups failures into three kinds. Temporally, models mishandled relative change, discontinued treatments and historical references such as a lesion previously graded LR-4 that has since evolved, sometimes extracting both the old and current grade as separate lesions. Syntactically, they misattributed treatment dates to nearby procedures or double-counted lesions summarised across multiple scans in one note. Clinically, they failed to infer implicit normalcy, for example reading a normal spleen as not reported when the report said normal solid abdominal organs.

The direction of error matters for downstream use. Hallucinated lesions improperly upstage disease severity, while omissions can hide secondary findings; the paper treats added events as the more severe failure. The practical priority that follows is not simply choosing the biggest model. Constrained decoding buys reliable structure at roughly 1.7 times the latency, the 70B general model beats the medical fine-tune, and the smaller 8B model plus regex remain viable where hardware or data-leakage constraints rule out larger deployments. For timeline work specifically, the paper's own evidence says to validate at the event level and to trust standardised transplant assessment reports over free-form radiology notes.

Sources used in this answer