[HealthSec 2025] Goal-Driven Risk Assessment: Securing the LLM-Powered Clinical Workflow

TL;DR

As Large Language Models (LLMs) move from simple chatbots to autonomous "Orchestrators" in healthcare, they introduce a terrifying new attack surface. This paper moves beyond simple threat lists to provide a goal-driven risk framework using attack trees. It identifies how a simple prompt injection can escalate into a catastrophic misdiagnosis (Risk Score: 20/25) by exploiting the gaps between AI reasoning and traditional system logic.

Positioning: This work bridges the gap between qualitative AI security taxonomies (like OWASP Top 10) and the rigorous, quantitative safety requirements of the healthcare industry.

Problem & Motivation: The "Abstract Threat" Trap

Traditional cybersecurity tools are excellent at spotting a SQL injection or a weak password. Modern AI security lists (like MITRE ATLAS) are great at identifying "Prompt Injection." However, in a medical setting, knowing a threat exists isn't enough. System designers need to know: How does a prompt injection actually result in a patient getting the wrong medication?

The authors argue that current models are too abstract. They treat threats as isolated events rather than Attack Paths. In a complex healthcare system—where an LLM might be calling external APIs, searching EHRs, and managing session memory—a single vulnerability can have cascading effects that lead to life-threatening outcomes.

Methodology: The Architecture of an Attack

The researchers modeled a representative healthcare system consisting of five core layers: a Web App, a Healthcare Platform (EHR), an Orchestrator (the LLM agent), External Resources (Translators/Databases), and the LLM itself.

The Core Mechanism: Attack Trees (AT)

Instead of listing what might go wrong, the authors started with what an attacker wants to happen (the Goal) and worked backward to find every possible path to get there.

The framework decomposes goals (G) into Risk Instances (R). For example, Goal G1 (Intervening in Medical Procedures) is broken down into:

• G1-R1: Misdiagnosis of Critical Illness.
• G1-R2: Unauthorized Procedures.
• G1-R3: Corrupted Medication Recommendations.
• G1-R4: Cross-Patient Context Contamination.

Methodology: The Core Anatomy of Risk

The study introduces a sophisticated scoring system for Likelihood, moving beyond "low/medium/high." It evaluates:

1. Business Rule Knowledge: How much clinical logic does the attacker need to know?
2. Technical Complexity: How hard is the actual exploit?

The Orchestrator emerges as the most sensitive control layer. Since it manages the "Task Planner" and "Task Executor," an error here allows an attacker to bypass safety checks (like allergy screenings) by injecting malicious tasks into the pipeline.

Experiments & Results: The High Stakes of AI in Medicine

The results for Goal G1 reveal a sobering reality. Prompt Injection is the "path of least resistance." Because it requires low technical complexity and no medical knowledge, its likelihood is rated 4 (Likely). When aimed at causing a misdiagnosis, the impact is 5 (Catastrophic), resulting in a maximum risk priority.

Critical Findings:

• Direct vs. Indirect Paths: Prompt injection is a Direct path to misdiagnosis. However, LLM Session Mismanagement is a Situational path—dangerous because it can lead to "Cross-Patient Context Contamination" (G1-R4), where a doctor sees one patient's data while treating another.
• The Insider Advantage: While external prompt injections are easy, "Model Tampering" and "Orchestrator Errors" become significantly more likely (Likelihood jumps from 2 to 3) if the attacker is a malicious insider with knowledge of the fine-tuning pipeline.

Deep Insight & Conclusion

This paper serves as a wake-up call for AI developers. The takeaway is clear: Context is everything. A prompt injection in a movie recommendation engine is a nuisance; in an LLM agent connected to a hospital’s infusion pump, it is a lethal weapon.

Future Outlook: The authors suggest that the next frontier is using LLMs themselves to generate these attack trees, potentially automating the discovery of vulnerabilities that human experts might miss. However, the reliance on high-quality sanitization and session isolation remains the immediate technical hurdle for "secure-by-design" AI systems.

Final Takeaway

Risk assessment for LLMs can no longer be a checklist. It must be a map—specifically, an attack tree map—that accounts for the unique emergent behaviors of large language models combined with the rigid complexities of clinical workflows.