Can AI extract breast cancer trial data from EHRs accurately?

What MIRROR actually compared

MIRROR is a retrospective comparison, not a prospective deployment. It took the electronic case report form (eCRF) data that trained clinical research staff had already entered during 11 breast cancer clinical trials and treated those entries as the reference standard. It then asked an AI tool called CapTrial to pull the same variables out of the hospital's electronic health records (EHRs) for the same patients, and measured how closely the two agreed.

The cohort was 113 breast cancer patients, all enrolled at the Virgen del Rocío University Hospital in Sevilla, Spain, which participated in all 11 trials. The tool processed 138,454 structured records and roughly 1,850,000 words of unstructured text. Eighteen data domains were assessed, spanning demographics, lifestyle factors, prior disease, receptor status, study treatment, concomitant medication, laboratory tests, radiological response and survival.

Because the eCRF was produced by trial staff working under Good Clinical Practice, the comparison asks a practical question: can automation reproduce what a trial site already records, rather than whether AI can discover something new. That framing matters for interpreting every number below.

Why the metrics differ by data domain

The paper does not apply one metric everywhere, and readers should not average across domains. For categorical variables with a defined positive class, such as gender, smoking status or receptor status, the authors computed true positives, false positives, false negatives and true negatives, then derived accuracy as (TP+TN)/(TP+TN+FP+FN), precision as TP/(TP+FP), recall as TP/(TP+FN) and F1 as 2TP/(2TP+FP+FN).

For record-level domains such as study treatment, concomitant medication and laboratory tests, each record is a unique combination of fields, so no meaningful positive or negative class exists. There, accuracy means the proportion of records in which every extracted parameter exactly matched the eCRF entry. For continuous or date variables such as overall survival and treatment start date, accuracy means exact match with the reference value.

A third case is current cancer: because every patient had confirmed breast cancer, there was no negative class, so accuracy was simply the proportion of cases where the AI prediction matched the eCRF. These are different denominators answering different questions, and a high figure in one does not transfer to another.

Where extraction was nearly exact

The strongest agreements came from structured or consistently formatted data. Administrative gender was available for all patients and matched with 100% accuracy. Birth date was present for 112 patients, representing 99% of records; the tool matched 78% exactly to the year, month and day, and the remaining birth years were recovered with 100% accuracy once a plus or minus one year window was allowed for partial dates or possible entry errors.

Concomitant medication showed the effect of denominator choice clearly. The trial database held 526 records from 93 patients (82%), while the AI tool retrieved 10,205 records from 109 patients (96%). Restricted to patients with data in the trial database, extraction accuracy was 100%.

Survival status behaved similarly. Death dates were present for 17 patients (15%) in the trial database, but the tool retrieved death dates for 27 patients (24%). After restricting to dates not posterior to each trial's end-of-study date, accuracy was 100%. In both cases the tool found more records than the manual reference, so the headline accuracy depends on which patients and records enter the comparison.

The 90% to 99% band and its denominators

Several domains landed between 90% and 99% accuracy. Menopausal status was recorded for 67 patients (59%) in the trial database, and 61 of them (91%) were matched correctly. Drug use was collected for 64 patients (57%); the tool retrieved information for 42 patients (37%), and 26 patients (23%) matched in both databases, of whom 25 (96%) matched accurately.

Study treatment was retrieved for 81 patients (72%). The authors manually inspected the EHRs of patients with no data in the real-world database and confirmed that no mention of the trial treatments had been recorded there. Restricting the analysis to patients with anticancer treatment data in their EHRs produced 99% accuracy.

Current cancer type was extracted for all patients, and with a large language model used to assess equivalence of medical terms, the reported accuracy was high. The recurring pattern is that the tool often found documentation the trial database lacked, and the reported accuracy rises when the comparison is limited to patients with source documentation.

How the pipeline handles structured and narrative text

CapTrial is not a single model. Variables were primarily taken from structured EHR tables, such as laboratory tests, pharmacy records, birth date and overall survival, using extract, transform and load (ETL) processes that consolidate sources into a unified repository. Regular expressions handled consistently formatted patterns.

For partially structured, inconsistently coded or narrative variables, the pipeline selected relevant documents within each trial's observation window, preprocessed free text through tokenization, sentence segmentation and section detection, then detected candidate entities with regular expressions and medical entity recognition. Pretrained BERT-like encoder models were used for token-level recognition, including detecting negation, without modifying their pretrained weights.

A GPT-4-based prompt was used in a zero-shot manner for domains needing contextual inference across notes, such as prior cancer or non-cancer disease, and to aggregate or normalize selected outputs. The authors report that each method performed differently by data type: ETL and regular expressions for structured data, medical entity recognition for narrative mentions, and the language model for context-dependent inference.

Traceability and the absence of task-specific training

Every extracted variable came with references to the original EHR text, and raw data were retained for mismatches. Reviewers could inspect discordant cases and judge whether a difference came from an extraction error, missing or incomplete EHR documentation, or an error or inconsistency in the eCRF. The authors describe this as a validation procedure that improved data quality and reduced checking time.

No task-specific model training used the MIRROR cohort. The system relied on pretrained components, so no training dataset was built from these patients. A development subset of at most 10% of available data was used in selected tasks to adjust extraction pipelines, and performance was evaluated on the remaining 90% to 100%, depending on the domain.

Formal interannotator agreement was not calculated, because no parallel independent human annotations were generated. The reference standard is therefore routine eCRF practice, not a second blinded abstraction, which limits how much of a discrepancy can be attributed to the AI versus the original manual entry.

What the authors prioritize next

The authors summarize the result as at least 80% accuracy, precision, recall and F1-score for most clinical domains, and they place unstructured data extraction and standardized eCRFs across sites at the top of the next round of work. The reason is visible in the results: narrative fields such as menopausal status, drug use and prior disease depended on explicit documentation, and absence of documentation was allowed in the domain definitions.

The practical reading is a split by data type rather than a single verdict on AI extraction. Structured and consistently formatted fields, including gender, birth year, concomitant medication and survival dates, approached exact agreement once denominators were aligned. Narrative and context-dependent fields produced lower or more conditional agreement, and the paper's own recommendation is to improve those and to standardize eCRFs so that extraction and evaluation are consistent across sites.

The most decision-relevant threshold is therefore not one accuracy number but the match between extraction method and data type, plus a clear rule for what counts as the reference when the EHR contains more documentation than the trial database. MIRROR supports using AI extraction where structured sources exist and where traceable text references allow verification, and it flags narrative domains and cross-site eCRF variation as the places where that support is weakest.

Sources used in this answer