Can multi-agent AI support liver disease guideline development and maintenance?

A seven-agent panel answering 137 binary questions

The paper asks whether a multi-agent AI system can take over part of the labor of building and updating clinical guidelines. The authors built MARG, a Multi-Agent Retrieval-Augmented Generation system, and tested it on a question bank drawn from seven 2025 European Association for the Study of the Liver guidelines. The bank, called LiBank, contains 137 clinical scenarios.

Retrieval-augmented generation means the model fetches external documents at answer time and grounds its output in them, rather than relying only on what it learned during training. MARG extends this to seven separate agents — AI4S Pro, ChatGPT GPT-4 Omni, Consensus Pro, Copilot, DeepSeek, Gemini 2.5 Pro and MetaSo 0.99 — chosen because each was publicly accessible, had a retrieval-enabled deep-search mode, exposed identifiable literature sources, and produced output structured enough to extract recommendations and evidence grades.

The question bank was deliberately narrow. Only questions answerable with a binary judgment were kept, such as those beginning "Should", "Is/Are", "If" or "Can". Open-ended questions, region-specific content and recommendations without clear directionality were excluded. Each question then went through a four-step refinement: splitting multi-part questions, neutralizing phrasing to reduce response bias, adding clinical context, and standardizing terminology.

Each agent followed a fixed prompt sequence: make an initial yes/no judgment, retrieve and tabulate the latest literature while excluding guidelines and consensus statements to prevent recommendation leakage, grade the evidence on the Oxford Levels of Evidence hierarchy used by the 2025 EASL panels, then revisit the initial answer in light of the synthesized evidence.

Agreement rises sharply as expert consensus rises

Overall, MARG agreed with the published guidelines in 77.4% of the 137 scenarios. That single number hides the most decision-relevant pattern in the paper: agreement tracked how settled the underlying recommendation was. Where the guideline panel itself had only weak consensus, consistency was 45.8%. At full consensus it reached 87.8%.

Performance also varied by the type of clinical question. Concordance was 65.5% for diagnosis, 75.0% for prevention and 81.0% for treatment. Diagnosis is the weakest domain, and it is also where the paper's own worked examples show the system diverging from guidelines on imaging choices.

This matters for how the number should be read. A 77.4% average is not a uniform accuracy figure that applies equally to every recommendation. It is an average over scenarios whose difficulty varies systematically, and the system is closest to the guideline when the guideline itself is least contested.

The paper's own explanation is that the agents show a metacognitive deficit: their self-described reasoning does not match their actual performance, and retrieval augmentation does not fix it. That is the authors' interpretation of the pattern, not a separately measured mechanism.

Newer evidence does not automatically mean better answers

A separate part of the evaluation compared the evidence the agents retrieved with the evidence cited by the guideline panels. Here MARG looked stronger. It outperformed manual review on recency (6.3% versus 0.9%), reproducibility (25.5% versus 7.5%) and quality (24.2% versus 8.0% Level I evidence).

These are different measures from the concordance figure and should not be merged with it. Concordance asks whether the system's yes/no answer matches the guideline. The recency, reproducibility and quality percentages describe properties of the cited evidence base. A system can retrieve fresher, higher-grade studies and still reach a different conclusion than the guideline panel.

The paper's worked examples make that concrete. On whether contrast-enhanced ultrasound should be first-line for noninvasive diagnosis of hepatocellular carcinoma, the guideline says no, citing computed tomography or magnetic resonance imaging as preferred. MARG answered yes, citing studies reporting that contrast-enhanced ultrasound performs comparably. On whether selective internal radiation therapy is an acceptable alternative to systemic therapy in advanced disease with segmental or lobar portal vein invasion, the guideline says no; MARG said yes, citing trials reporting noninferiority to sorafenib.

The authors present these as guideline evidence challenges or controversies that the expert panel validated, not as straightforward errors. That framing is the paper's, and it is the point where a reader should be most careful: a disagreement can reflect an outdated guideline or a system that has weighted the literature differently, and this study design cannot fully separate the two.

Structured debate resolves most conflicts among agents

The agents did not simply vote. They went through four rounds of structured debate, and the paper reports that this resolved 95.8% of recommendation conflicts, with strong consensus above 85% in 74.5% of final outputs.

Agreement among agents was only moderate, however: Fleiss' kappa of 0.459 for clinical recommendations. Human-AI concordance on evidence grading was weaker still, with a weighted kappa of 0.192. Kappa here measures agreement beyond what chance would produce, so these values describe how far the agents were from acting as independent, consistent raters.

The paper also documents failure modes that emerged during debate. It describes herd behavior, where secondary agents shifted evidentiary positions toward the majority view; artificial neutrality, where agents produced excessive "all perspectives have merit" responses in contentious scenarios; and protocol noncompliance, where outputs were overly verbose and ignored formatting constraints.

These behaviors qualify the conflict-resolution figure. Resolving 95.8% of conflicts means the agents converged, not necessarily that they converged on the correct answer. The authors connect the pattern to a tension between conversational model architectures and the evidentiary standards medical decisions require.

The human-AI simulation shows a gain concentrated early

The human side of the evaluation was a small simulation: 60 scenarios assessed by physicians with 5, 15 and 30 years of experience. After interacting with the system, the share of guideline-inconsistent decisions corrected was 60%, 40% and 50% respectively. Deviations induced were 10%, 10% and 20%, giving net gains of 50%, 30% and 30%.

The abstract summarizes this as correction of 40–60% of guideline-inconsistent decisions, with the 50% net gain most apparent in the early-career tier. The per-physician table is the more precise source: the 5-year physician had the largest net gain, while the 30-year physician corrected as many deviations as the 15-year physician but induced twice as many.

Before interaction, agreement with guidelines was low across the board. Median kappa was 0.287 for the 5-year physician, 0.288 for the 15-year and 0.219 for the 30-year, against guideline-panel values of 0.165, 0.158 and 0.302. The paper also flags potential automation bias, meaning risk of AI-induced deviation from best clinical standards, as something observed during validation.

With 60 scenarios and one physician per experience tier, this is a signal about direction, not a measured effect size for clinical practice. The authors describe it as a small-scale simulation.

Where the evidence stops

The evaluation was conducted against existing guidelines rather than prospectively in real-world clinical settings. The authors state that trust in retrieval accuracy and reasoning reliability cannot be fully assured without prospective validation.

The study did not assess impact on clinician workload, user acceptance or cognitive engagement, all of which the authors call critical for integration into practice. It also did not evaluate downstream patient outcomes, including treatment appropriateness, safety or clinical effectiveness. Whether the corrections or newly surfaced evidence translate into patient benefit remains unknown.

Generalizability beyond liver disease, or beyond the EASL guideline development framework, is untested. The question bank was also restricted to binary questions from 2025 EASL guidelines, chosen because their standardized methodology provides consistent benchmarks.

The tiered collaboration model is the paper's real proposal

The paper's conclusion is not that the system should write guidelines. It proposes a tiered human-AI collaboration model in which humans remain decision-makers and AI acts as an accelerator, with expert oversight reserved for predefined conditions.

In that model, clinical leads and guideline methodologists first define the scope of questions, approved evidence sources, grading rules and escalation triggers. MARG then runs routine evidence surveillance, retrieval, preliminary synthesis and flagging of disagreements among agents. Outputs are split by risk: high-risk outputs — low-consensus findings, major changes in recommendation direction or strength, safety signals, or findings relevant to underrepresented populations — go to senior specialists or the full committee, while moderate-risk outputs get rapid review.

The authors attach four prerequisites before any task is delegated: globally accessible evidence repositories that retain population, treatment and setting characteristics; a consortium producing machine-interpretable grading protocols aligned with frameworks such as GRADE or the Oxford Levels of Evidence; guideline-specific benchmarks measuring consistency, coverage of uncommon high-stakes conditions, and update latency after landmark trials; and independent ethical governance with disclosure of model and database versions, search dates, retrieved and excluded sources, conflicts of interest and audit logs.

The practical priority that follows from the evidence is a split by consensus level and risk rather than uniform automation. The system's agreement with guidelines was highest where expert consensus was already full (87.8%) and lowest where it was weak (45.8%), and the largest simulated net gain appeared in the least experienced tier. Those are the conditions under which the paper's own numbers support leaning on the system, and the conditions under which it should escalate to people.

Sources used in this answer