[Theoretical Insight] Toward Bayesian AI: Overcoming Over-reliance by Modeling Base Rate Neglect

TL;DR

Blindly following AI advice is not just a "trust" issue—it's a mathematical failure in human reasoning. This paper argues that over-reliance is actually base rate neglect. By redesigning AI interfaces to force users to provide "priors" and then explicitly calculating "posteriors" using Bayes' Theorem, we can turn passive AI-users into rational Bayesian agents.

Positioning: This work moves beyond common "explainable AI" (XAI) tropes. It is a theoretical bridge between Cognitive Science (Kahneman's heuristics) and HCI, proposing a normative mathematical framework for human-AI collaboration.

The Problem: The "Trust" Trap

In high-stakes fields like medicine or law, we want AI to assist, not replace. However, humans often exhibit automation bias: they follow the machine even when their eyes tell them otherwise.

Current HCI research tries to fix this by "calibrating trust." The logic is: "If the AI is 80% accurate, the human should trust it 80% of the time." The authors argue this is insufficient. Even if trust is calibrated, users often fail to synthesize the AI's output with external information (the "base rate"). When a hiring manager ignores that 90% of employees in a sector leave within a year because a specific AI says this candidate is a "star," they are committing a classic cognitive error.

The Insight: Over-reliance = Base Rate Neglect

The authors propose that over-reliance is a deviation from Bayesian Rationality.

In a rational world, when you see AI evidence ($E$), your belief in a hypothesis ($H$) should follow Bayes' Rule:

$$p(H|E)=\frac{extRecall\cdot p(H)}{extRecall\cdot p(H)+extFPR\cdot p(egH)}$$

The failure occurs in two ways:

1. Uniform Prior Assumption: The user acts as if $p(H)=0.5$, ignoring the actual prevalence of the event.
2. Inverse Fallacy: The user confuses the AI's reliability (e.g., "The AI is 90% accurate") with the probability that the hypothesis is true given the AI said so.

Methodology: The "AI-Follow" Interface

To fix this, the authors suggest a prototype interaction flow designed to "force" Bayesian thinking:

1. Step 1 (The Prior): The user reviews the case without AI and provides a preliminary prediction and a probability estimate ($p(H)$).
2. Step 2 (The Evidence): The AI provides its suggestion, explicitly showing its reliability metrics (Recall and False Positive Rate).
3. Step 3 (The Synthesis): The system computes the posterior probability and provides a slider for the user to see how sensitive the result is to their original prior.

Figure 1: The proposed interface. Left: User sets the prior. Middle: AI provides evidence + reliability. Right: The Bayesian posterior calculation.

Why This Works (and Where it Fails)

By making the math explicit, the interface serves as a Cognitive Forcing Function. It stops the user from using the AI suggestion as a mental shortcut (heuristic) and instead forces a "System 2" (slow, analytical) reasoning process.

Key Results & Expectations:

• Mitigation of Bias: By asking for "probability" instead of "confidence," the system avoids the "alignment" trap where human confidence simply mimics AI confidence.
• Robustness Check: The interactive slider (Figure 1, Right) allows the user to see: "Even if I think the prior is low, how strong must the AI be to change my mind?"

The core mathematical driver for the interaction: Integrating Recall and FPR with Human Priors.

Critical Perspective

The primary limitation is the cognitive load. Requiring experts to provide numerical probabilities for every case is time-consuming and difficult for humans who aren't trained in statistical thinking. While the paper suggests "Frequency Trees" as a potential solution, the friction of this interaction might lead to different types of user fatigue.

Future Outlook

This paper is a call to action for AI developers to stop treating AI as a "black-box truth teller" and start treating it as "probabilistic evidence." As the EU AI Act begins requiring "meaningful human oversight," frameworks like this will be essential to prove that the human is actually thinking, not just clicking "Approve."