Can an audit-ready EHR foundation model improve order-time medication-safety auditing?

What the failure-to-alert rate actually counts

The paper's confirmatory endpoint is the reference-standard-defined order-time failure-to-alert rate. Its denominator is every inpatient medication order inside the high-risk auditing scope: renally cleared drugs (127 drug classes), drug–drug interactions with DrugBank severity of at least 2, and FDA-label contraindications. Its numerator is the eligible tier-i/ii reference-positive orders for which a system emitted no actionable alert at its own frozen operating point. The result is expressed per 1000 eligible orders.

That scope covered 847,000 orders across 312,000 patients and 89,000 encounters. The reference labels were frozen before final scoring, and each system was scored at a fixed operating point rather than tuned after the fact. The authors state plainly that this offline process metric does not condition on pharmacy verification, dispensing, medication administration, adverse drug events, or patient harm, so a lower miss rate means better agreement with the audit reference, not demonstrated clinical benefit.

A second measure is easy to confuse with the first. The false-alert rate is defined as false positives divided by final negative reference labels, and it counts alerts on orders labeled negative. It is not the same as a pharmacist override rate; logged override dispositions are reported descriptively but are explicitly not the mathematical definition of a false positive.

Stale and absent evidence is treated as a signal

The motivating problem is that real-time auditing needs current laboratory and medication history, while the record may hold values that are stale, imported from outside, or simply never measured. Treating an unmeasured value as normal is, in the authors' framing, a structural error rather than a benign imputation choice.

Clinical Trajectory Language is the representation built around that problem. Each patient record becomes a time-ordered sequence of events, and each event is broken into tokens for event type, time (including gaps between events and time relative to an anchor), coded attributes such as ICD-10, SNOMED CT, LOINC and RxNorm, optional numeric values with units, provenance (structured versus extracted, in-network versus external), and an explicit observability state.

The observability tokens name the missingness modes directly: MEASURED, NOT MEASURED, EXTERNAL ONLY, LOW CONFIDENCE, UNIT AMBIGUOUS, and STALE. The paper's creatinine example shows how this works in practice: a value of 2.1 mg/dL imported from an outside facility and more than 30 days old relative to the decision point carries both an external provenance token and a stale observability token. Staleness thresholds are analyte-specific, with creatinine at 30 days and INR at 7 days, and NLP-extracted values below 0.85 probability are marked low confidence.

ClinicalGPT is a 7B-parameter decoder-only transformer trained on these sequences with a 32,768-token context. Its pretraining mixes clinical record completion with an auxiliary observability head, next-order and care-event prediction, time-to-complication modeling, observability-aware denoising using empirically estimated workflow gap patterns, and an incremental clinical consistency objective aimed at reducing unstable recommendation reversals.

ClinicalGPT led on misses and false alerts together

ClinicalGPT with hard constraints reached 3.2 misses per 1000 eligible orders (95% CI, 2.8–3.7). Rules only reached 9.4 (8.6–10.3), the original generic LLM plus rules comparator reached 6.1 (5.5–6.8), and the strongest observed LLM comparator, RAG + LLM + rules, reached 4.8 (4.2–5.5). The paper reports these as relative reductions of 66% versus rules only and 48% versus the original generic LLM, and 33% versus the RAG comparator.

The input-matched control matters for interpretation. CTL-LLM plus rules used the same CTL input, observability tokens, training cohort, compute budget, prompt, hard-rule layer and suppression policy as ClinicalGPT but only token-level self-supervision, and it reached 5.1 misses per 1000 (4.5–5.8). That makes the gap between 3.2 and 5.1 the cleaner test of the workflow-aligned objectives rather than of input format alone.

The false-alert side moved in the same direction. ClinicalGPT plus rules produced 33.9 false alerts per 1000 orders (95% CI, 31.6–36.3), against 38.7 (36.2–41.3) for rules only and 46.3 (43.1–49.6) for RAG + LLM + rules. At the frozen operating points, the authors describe ClinicalGPT as occupying the favorable lower-left region of the safety–burden plane relative to every reported primary-endpoint comparator, meaning fewer misses and fewer false alerts at once.

Observability modeling and rules each add something

Matched ablations isolate the components. The authors trained variants that removed observability tokens, kept observability tokens but removed observability-aware denoising, replaced the workflow-conditioned corruption distribution with uniform random corruption, and used token-level self-supervision only. These controls shared architecture, training data and compute except for the named component, and the paper reports that they support the contribution of explicit observability modeling.

The rule layer was evaluated separately through three configurations: rules only, ClinicalGPT model only, and ClinicalGPT model plus rules. The model-only configuration outperformed rules alone, but the hybrid system achieved the lowest miss rate, which the authors read as a safety floor created by non-negotiable constraints on contraindications, severe interactions, dose bounds, and tier-i/ii deferral.

One caveat belongs with the rules comparison. The offline reference standard, the rules-only comparator, and the hard-constraint layer draw on related medication-safety source families, though they serve different roles and the reference standard never generated a deployed alert. Because that overlap could introduce incorporation bias, the authors interpret the rules-only comparison as an operational comparison under a shared safety ontology rather than a clean independent test.

Independent relabeling narrows the RAG gap

To probe that bias, the authors sampled 1200 orders into 48 strata crossing 12 medication groups with four decisive-evidence patterns: complete or current, missing, stale, and external-only or unit-ambiguous. Three board-certified pharmacists labeled the randomized, right-censored context while blinded to candidate labels, knowledge-base severity outputs, system identity and outputs, primary reference labels, and post-order events, with a fourth senior pharmacist resolving splits and uncertain cases.

On this enriched, knowledge-base-independent subset, sensitivity ordering was preserved: ClinicalGPT at 96.9%, RAG + LLM at 92.7%, and rules only at 88.5%. The ClinicalGPT–RAG gap, however, narrowed to 4.2 percentage points. The authors say this reduces but does not remove concern that the primary comparison is explained by shared knowledge sources. Because the sampling was deliberately enriched and not prevalence-representative, the paper reports raw confusion counts and conditional sensitivity and specificity rather than a population miss rate per 1000 orders.

A silent pilot measured workflow, not outcomes

A separate 12-week prospective silent assessment ran at one internal 847-bed site covering 124,000 medication orders. ClinicalGPT ran in parallel with the existing computerized provider order entry alert system without altering prescribing, dispensing, pharmacist actions, or other care, and research alerts were shown to 12 designated study pharmacists only after order signing and irreversible logging of the CPOE action.

Under a cohort-specific label vector frozen before linkage to either alert stream, the descriptive counterfactual miss rate was 2.9 per 1000 orders (95% CI, 2.1–3.8) for ClinicalGPT plus rules and 8.7 (7.2–10.4) for existing CPOE alerts. Median research-review time among flagged orders was 28 seconds (IQR, 19–41) versus 52 seconds (38–74). Pharmacists rated 89% of ClinicalGPT-generated evidence citations as helpful or very helpful versus 61% for existing alerts, and 78% versus 34% said they would have acted on the research alert.

These are workflow and counterfactual process measures from a single site, silent, non-randomized and non-interventional. They show the system can run prospectively and that research alerts can be reviewed within observed workflow times; they do not show deployed clinical effectiveness, prevented adverse drug events, or causal patient benefit.

Where the comparison stops transferring

The evaluation corpus spans 2015–2024 across seven health systems, with four internal sites (2.1M patients) used for development and three external sites (1.1M patients) held out for evaluation. The primary analysis pools the seven-site estimate, and individual-site denominators, event counts, rates and confidence intervals appear in supplementary tables rather than in the headline number.

Temporal drift was tested separately on a secondary 18-task benchmark under a strict time split, not on the primary miss rate. From a 2021-04-01 training cutoff through month 36, ClinicalGPT declined 5.0 macro-score points versus 12.0 for the selected comparator, and event-triggered recalibration produced the lowest month-36 expected calibration error at 0.019, versus 0.022 for scheduled quarterly recalibration and 0.036 for fixed temperature. The authors describe this as relative robustness, not absence of drift.

Deployment cost is also part of the picture. The 7B model ran at 142 ms median and 218 ms p95 latency with 7.0 orders per second on an A100 80GB, while a distilled 1.3B variant ran at 47 ms median and 21.3 orders per second with a 3.4 per 1000 miss rate. Model weights remain non-public, and the datasets are not publicly available because of patient privacy and multi-institutional governance restrictions.

The priority is the paired miss and false-alert position

The decision-relevant claim is not that a foundation model beats rules in the abstract. It is that at frozen operating points, a hybrid of learned reasoning over observability-aware trajectories plus hard constraints landed at 3.2 misses and 33.9 false alerts per 1000 eligible orders, improving both sides of the trade-off relative to rules only at 9.4 and 38.7, and relative to the strongest observed LLM comparator at 4.8 and 46.3.

The honest reading of the sensitivity analysis is that part of the advantage over retrieval-augmented LLM baselines depends on the labeling source: independent de novo labeling kept the ordering but cut the ClinicalGPT–RAG gap to 4.2 percentage points on sensitivity. Readers weighing this system should therefore treat the rules-only comparison as an operational one under shared knowledge sources, and treat the input-matched CTL-LLM comparison at 5.1 misses per 1000 as the sharper evidence that the training objectives, not just the input format, carry part of the gain.

What remains unmeasured is the step from an order-time audit to patient outcomes. The paper's own boundary is that the endpoint is an offline proxy and that potential-harm tiers are counterfactual severity classifications rather than observed injuries, so the case for deployment rests on comparative auditing performance and workflow feasibility, not on demonstrated harm reduction.

Sources used in this answer