[ICLR 2026] From Narrow to Panoramic Vision: Reshaping Multimodal Cold-Start via Attention Guidance
From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning
The paper introduces AVAR (Attention-Guided Visual Anchoring and Reflection), a cold-start framework for Multimodal Large Reasoning Models (MLRMs) that optimizes the "Visual Attention Score" (VAS). By reshaping how models allocate attention between system and visual tokens, the framework achieves a 7.0% average performance gain across 7 benchmarks, establishing a new SOTA for 7B-scale models.
TL;DR
Researchers have discovered a counter-intuitive "Lazy Attention Localization" phenomenon: most multimodal models don't actually "look" at images more when trained on multimodal data. By introducing AVAR, a framework that explicitly penalizes system token obsession and rewards visual anchoring, the authors boosted Qwen2.5-VL-7B performance by 7.0% on average, including a massive 12.2% jump in geometric reasoning.
The "Narrow-View" Bottleneck: Why More Data Fails
In the race to build Multimodal Large Reasoning Models (MLRMs), the "cold-start" (initialization) phase is often treated as a black box. The prevailing mystery was why text-only initialization often outperformed multimodal training.
The authors quantify this using a new metric called Visual Attention Score (VAS). They found a near-perfect correlation (r = 0.9616) between VAS and reasoning performance. Their discovery? Most models are "Narrow-View"—they focus heavily on system prompts/formatting tokens and treat visual inputs as an afterthought. Alarmingly, multimodal cold-start training fails to move the needle on VAS, while text-only training accidentally helps. They call this Lazy Attention Localization.
Figure 1: (a) The undeniable link between Visual Attention Score and Benchmark Performance; (b) How text-only cold-start (OVR-CS) outperforms standard multimodal starts.
Methodology: Engineering "Panoramic Vision"
To break the "Narrow-View" habit, the AVAR (Attention-Guided Visual Anchoring and Reflection) framework intervenes at three levels:
1. Visual-Anchored Reflection Data Synthesis
Instead of simple "caption-then-reason" data, AVAR uses a 3-stage pipeline to create "thinking" chains that include explicit visual checks (e.g., "Looking back at the triangle in the image..."). This forces the model to treat visual inputs as a constant reference point.
Figure 3: Generating data that naturally enforces visual reflection and anchoring.
2. Attention-Guided Training Objectives (AGTO)
The authors didn't just rely on data; they modified the loss function. AVAR introduces a dual-loss objective:
- Image Enhancement Loss: Maximizes attention weights on visual tokens.
- System Suppression Loss: Minimizes attention weight "wasted" on repetitive system prompts.
This mathematically forces the model to reallocate its internal "budget" toward the image.
3. Visual-Anchored Reward Shaping (VARS)
During Reinforcement Learning (using the GRPO algorithm), the model isn't just rewarded for getting the right answer. It receives a "Visual Reward" based on its VAS during the reasoning process. Correct answers derived from visual grounding are worth more than correct answers derived from language priors.
Experimental Results: A New 7B SOTA
The results across 7 benchmarks demonstrate that AVAR effectively "wakes up" the model's eyes.
- MathVision: +12.2% (The model can finally "see" geometric relationships).
- HallusionBench: +8.8% (Significant reduction in hallucinations).
- MMMU: +5.7% (Better multi-disciplinary expertise).
Table 1: AVAR-Thinker outperforms all existing 7B reasoning models and even rivals much larger closed-source models.
Critical Insight: The "System Token Redundancy Zone"
One of the most profound takeaways is the identification of the System Token Redundancy Zone. By performing training-free interventions (simply scaling down attention to system tokens at inference), the authors achieved a 1-2% gain without any training. This proves that LLMs are "distracted" by their own system instructions, and effectively "muffling" these instructions allows the visual signal to shine through.
Conclusion
AVAR shifts the paradigm of multimodal training from simple alignment to attention hygiene. By proving that visual grounding is a behavior that can be incentivized through objective functions and reward shaping, this work provides a blueprint for the next generation of Reinforcement Learning-based vision models.
Limitations: While effective, the computational cost of calculating VAS-based rewards during RL is higher than simple accuracy rewards. Future work may need to find more efficient proxies for visual attention.