Can large language models detect early epilepsy from routine electronic health records?

Why seizure notes are hard to classify

Epilepsy and psychogenic non-epileptic seizures (PNES) can look alike when a patient is seizing, but they require fundamentally different management. The paper notes that 20–30% of patients referred to tertiary epilepsy centers for presumed drug-resistant epilepsy are ultimately diagnosed with PNES, so the distinction carries real consequences for treatment and morbidity.

The diagnostic gold standard, prolonged video-electroencephalography (vEEG), is costly and capacity-limited: the paper reports a single vEEG admission in the United States commonly exceeds $15,000, with typical waits of 8–12 weeks and monitoring lasting 1–5 days. That gap motivates a cheaper screen built from text that is already written down.

The text itself is the obstacle. Features such as ictal duration, eye closure, or tongue biting are statistically more common in one condition or the other, but none is pathognomonic, and phrases like "loss of consciousness" or "generalized shaking" appear in both. The authors argue this is why rule-based or feature-dependent models, and off-the-shelf LLMs, perform poorly: the signal lives in how the whole narrative is contextualized, not in isolated keywords.

How EpiScreen turns a note into a probability

EpiScreen is not a new architecture. The team fine-tuned four open-source LLMs — three general-domain models (Llama-3.3-70B, Qwen-2.5-32B, DeepSeek-R1-32B) and one medical model (Med42-70B) — on labeled clinical notes, using QLoRA with 4-bit quantization, a LoRA rank of 16, a learning rate of 5e-4, three epochs, and a maximum sequence length of 4096 tokens. No additional hyperparameter tuning was performed.

For each note, the model is prompted to choose between "Epilepsy" and "PNES". The predicted probability comes from the model's own confidence scores at its final prediction layer, normalized with log-softmax across the two logits. Inference uses fixed weights and deterministic score-based decoding, so the same input yields the same prediction in the same runtime environment.

The comparison set matters for reading the results. Baselines included FSLS, a phenotype-based rule system, plus BERT and ClinicalBERT trained as standard supervised models on the same data; BERT-based models were adapted to 4096 tokens by splitting notes into 512-token windows and mean-pooling. Zero-shot prompting was tested on the same general LLMs and on four medical LLMs (Med42-70B, MedGemma-27B, Baichuan-M2-32B, HuatuoGPT-o1-70B).

Fine-tuning beats prompting and rule-based baselines

Within the same institution, EpiScreen outperformed every baseline on both datasets. On MIMIC-IV, the best EpiScreen variant (Med42-70B) reached an AUC of 0.891 (95% CI 0.866–0.915) and accuracy of 0.832, versus 0.769 AUC for ClinicalBERT, 0.756 for BERT, and 0.695 for FSLS. On the UMN cohort, EpiScreen variants reached AUC 0.974–0.992, compared with 0.960 for ClinicalBERT and 0.944 for FSLS.

The gap between prompting and fine-tuning is one of the paper's clearest signals. Fine-tuning produced average absolute AUC gains of 13.2% on MIMIC-IV and 11.6% on UMN relative to off-the-shelf LLMs. Prompted Llama-3.3-70B, for instance, reached 0.813 AUC on MIMIC-IV but only 0.645 accuracy, while prompted MedGemma-27B fell to 0.725 AUC on MIMIC-IV and 0.799 on UMN.

AUC and accuracy are not interchangeable here, and the paper's own table shows why: a model can rank cases well while still misclassifying many at a fixed threshold. Readers comparing EpiScreen variants should look at both columns rather than treating the headline AUC as a deployment-ready accuracy figure.

Cross-institution transfer separates the models

The more demanding test trained on one institution and tested on the other. Here the baselines collapsed: FSLS trained on UMN reached an AUC of 0.498 on MIMIC-IV, a 47.2% relative decline from its in-domain performance, and BERT trained on MIMIC-IV reached 0.701 on UMN, a 26.6% reduction.

EpiScreen degraded far less. Trained on UMN and tested on MIMIC-IV, EpiScreen variants reached AUC 0.829–0.875; trained on MIMIC-IV and tested on UMN, they reached 0.966–0.980. The authors attribute this to the models abstracting higher-level semantics and relational patterns — such as links between triggers and prodromes — rather than anchoring on exact keywords, and to broad pretraining that normalizes differences in local documentation style.

That explanation is the authors' interpretation, not a directly measured mechanism. What the tables establish is the comparison itself: the cross-institutional AUC of 0.875 is the figure behind the abstract's headline, and it is lower than the in-domain numbers, so the honest reading is strong but not lossless transfer.

What the model reads in the note

To open the black box, the authors used sentence-level Integrated Gradients, which estimates how much each sentence shifts the prediction away from an empty baseline. Sentences with larger attribution magnitudes are treated as more salient, and the team aggregated them into 11 neurologist-curated phenotype categories, including seizure semiology, pre-ictal symptoms, post-ictal features, longitudinal temporal pattern, external triggers, and healthcare utilization pattern.

The reported case study shows the pattern the authors expect: the highest-scoring sentences describe witnessed event semiology, an aura preceding headaches, and a cluster of nearly ten events in one day, with attribution scores of 0.946, 0.930, and 0.894. The authors read this as evidence that EpiScreen weighs constellations of clinical features rather than single words, consistent with how neurologists reason.

This is explainability evidence, not proof of causation. The attribution scores show which sentences the model relied on; they do not show that those features cause the diagnosis, and the paper presents them for predicted epilepsy cases rather than as a validated clinical reasoning trace.

Clinicians plus EpiScreen versus clinicians alone

The collaboration experiment is the paper's most practice-relevant result. Eight clinicians formed four groups: three groups of two neurology residents averaging four years of training, and one group of two senior neurologists with over ten years of experience each. They reviewed 300 cases randomly selected from the MIMIC-IV test set with stratified sampling, had internet access, and decided by within-group consensus.

In this setting, EpiScreen-assisted neurologists outperformed unaided experts by up to 14.3%. The model output shown to clinicians was a case-level prediction plus sentence-level attribution scores indicating the textual evidence behind it, so the assistance was not a bare label.

The design bounds the claim. The comparison used 300 MIMIC-IV cases and consensus within small groups, so the 14.3% figure describes this simulated review setup, not measured patient outcomes, and it does not establish how much assistance helps in a live clinic with different case mix or time pressure.

Where the screening claim holds

Two sensitivity analyses address label trust. Restricting to 502 epilepsy and 177 PNES UMN test cases confirmed by long-term vEEG — the accepted gold standard — the same trained models were applied without retraining. A second analysis used 300 vEEG-confirmed UMN cases while retaining preliminary diagnostic suspicions such as "suspect epilepsy" without disclosing the final diagnosis, approximating the imperfect information available at first assessment.

The intended use is also explicitly staged. EpiScreen is designed to operate at the specialty-care stage, on narratives generated during and after neurology referral, to support earlier diagnosis before confirmatory video EEG. It is a screening aid built from notes that already exist, not a replacement for vEEG, and the cost and access argument in the paper rests on that sequencing.

The authors frame the payoff as timely, low-cost screening that may reduce diagnostic delays and avoid unnecessary interventions, particularly in resource-limited regions. That is a stated potential, not an outcome the study measured; the evidence supports discrimination between epilepsy and PNES from notes, plus a gain in a simulated clinician review.

The priority is transfer, not headline AUC

The number to carry away is not the 0.980 in-domain AUC on the private UMN cohort, which comes from data at the same institution where models were trained. It is the cross-institutional result: EpiScreen reached 0.875 AUC when trained on UMN and tested on MIMIC-IV, while FSLS fell to 0.498 and BERT to 0.701 in the same transfer direction.

That comparison is what makes a notes-only screen plausible across health systems, where documentation style, terminology, and detail vary. It also sets the realistic expectation: transfer costs accuracy relative to in-domain performance, and the clinician-assistance gain of up to 14.3% was measured on 300 MIMIC-IV cases in a consensus-based review, not in prospective care.

For a reader deciding whether this matters, the paper's own priority is clear: the value lies in a low-cost screen that holds up when institutions change, deployed before vEEG rather than instead of it, with the transfer gap — not the best in-domain score — as the figure that should shape expectations.

Sources used in this answer