Arbitration Failure, Not Perceptual Blindness: Why Your VLM Still Thinks a Blue Banana is Yellow
Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
This paper investigates why Vision-Language Models (VLMs) often succumb to linguistic priors (e.g., calling a blue banana "yellow"). Using a novel "Multimodal Arbitration Crossover" (MAC) analysis and activation patching across ten models (7B–72B), the authors demonstrate that models successfully encode visual counterfacts but fail during the "arbitration" phase, where linguistic biases override sensory input.
TL;DR
When a Vision-Language Model (VLM) looks at a blue banana and calls it "yellow," is it because it's "blind" to the color, or is it just "stubborn" due to its linguistic training? This paper reveals a striking Encoding–Grounding Dissociation: VLMs actually perceive the world correctly almost every time, but they fail to act on that perception. By analyzing ten models (from 7B up to 72B), the authors show that grounding errors are failures of internal arbitration, which can be partially fixed through training-free activation steering.
The Perception Myth
The common industry assumption has been that if a model hallucinates an object's property, the vision encoder probably missed the detail. This "Perceptual Blindness" theory suggests we just need better encoders or higher resolutions.
However, this research proves otherwise. By utilizing Logit Lens (reading internal "thoughts" layer-by-layer) and linear probes, the authors found that the "blue" in a blue banana is linearly decodable with over 86% AUC within the first 10% of the network's layers. Even in samples where the model eventually gives the wrong answer, the internal representation of the correct visual attribute is just as strong as in the samples where it gets it right.
Multimodal Arbitration Crossover (MAC)
How do we find where the model "changes its mind"? The authors introduce MAC analysis. It tracks the competition between the Visual Logit (what the model sees) and the Prior Logit (what the model knows from text).
Figure 1: The arbitration bottleneck. The vision encoder sees "blue," the LLM backbone processes "blue," but the final arbitration layer chooses the linguistic prior "yellow."
Key Findings in MAC:
- Crossover Layers: The point where the visual signal finally overtakes the linguistic prior varies significantly, occurring anywhere from 36% to 71% of the way through the model.
- Scaling Effect: Larger models (like Qwen2-VL-72B) commit to the visual answer earlier and with much higher confidence (wider logit gaps).
- Attribute Specificity: Models resolve "Color" and "Size" conflicts using different internal pathways.
Methodological Breakthrough: Full-Sequence Patching
In traditional NLP-based mechanistic interpretability, researchers often use "last-token patching." This paper identifies a critical flaw: In VLMs, last-token interventions are useless.
Figure 3: Why full-sequence patching matters. Visual information is distributed across all image tokens, not just the last text token.
By patching the entire sequence of tokens, the authors could flip the model's output from the prior ("yellow") to the visual truth ("blue") in up to 84% of cases. Crucially, they found that text tokens carry zero causal impact on these visual grounding decisions—the power lies entirely in the image token sequence.
Closing the Gap with Steering
Moving from diagnosis to cure, the authors tested two training-free interventions:
- Linear Steering: Adding a "visual direction" vector to the hidden states.
- SAE-Guided Steering: Using Sparse Autoencoders to target specific features precisely.
Table 3: Results of steering. Note that early-layer intervention (e.g., Layer 3 or 4) consistently outperforms MAC-layer intervention.
The most counter-intuitive finding? Intervention must happen early. If you wait for the MAC crossover (where the model "decides" on the visual answer), the internal weights are already too committed. Steering at the very first few layers allowed InternVL2 and Qwen2-VL to improve their grounding accuracy by nearly 4% with zero architectural changes.
Critical Insight & Future Work
The "Encoding–Grounding Dissociation" is a fundamental characteristic of current connector-based VLM architectures. Even at 72B parameters, the model's "brain" sees the truth but its "mouth" repeats a lie learned from text.
Takeaway for Practitioners: To solve hallucinations, don't just throw more pixels at the encoder. Instead, look at how the residual stream handles conflicts between modalities. The future of VLM robustness lies in more sophisticated arbitration mechanisms—ensuring that what the model "sees" is actually what it "says."
Limitations
The study focuses on synthetic counterfactuals (blue bananas). While this provides a perfect "laboratory" to isolate arbitration, real-world visual-linguistic conflicts may be more nuanced and require broader attribute testing.