[arXiv 2026] Routing Distraction: Why MoE Models See the Image but Forget to Think

TL;DR

Even when a Multimodal MoE model "sees" every number in a math problem correctly, it often fails to solve it—yet it succeeds if you just send the same problem as text. This paper identifies Routing Distraction as the culprit: visual inputs lure the model into using the wrong "experts" in its internal architecture. By nudging the model to stay focused on reasoning experts, the authors recover lost performance across SOTA models like Qwen3 and Kimi-VL.

The "Seeing but Not Thinking" Paradox

In the world of Vision-Language Models (VLMs), we usually blame failures on "perception errors" (the model didn't see the text in the image) or "alignment issues" (the model doesn't understand that the picture of a '3' means the number 3).

However, this paper uncovers a more frustrating failure mode. Through controlled experiments rendering math problems as high-res images, the authors found that models like Qwen3-VL-30B accurately extract 100% of the information but still mess up the logic. When the same logic is tested via text, the model is perfectly capable.

The Insight: The problem isn't the brain's "knowledge"; it's the brain's "routing."

Why Does This Happen? The Routing Distraction Hypothesis

The authors performed a "medical checkup" on the MoE (Mixture-of-Experts) layers and found a structural divide:

1. Visual Experts cluster at the beginning and end of the model (perception and output formatting).
2. Domain Experts (the "logic" units) live almost exclusively in the middle layers.
3. The Divergence: When an image enters the system, the "Router" (the traffic cop of MoE) gets distracted by the visual modality. In those critical middle layers, the image input forces the model to choose different experts than the text input does—often bypassing the logic experts entirely.

Above: Note how Math experts (red) are trapped in the middle, while Visual experts (blue) take over the early and late stages.

The Fix: Routing-Guided Intervention

Since we know where the logic experts are, why not just force the model to use them? The authors proposed a Soft Intervention strategy:

• Step 1: Identify "Domain Experts" by seeing which ones fire up when the model processes a small set of text-based math problems.
• Step 2: During visual tasks, slightly boost the "votes" (logits) for these identified experts in the middle layers.

Unlike "Hard Intervention" (which forces an expert on the model), "Soft Intervention" lets the model maintain some flexibility while ensuring the logic units aren't ignored.

Experimental Battleground

The team tested this across massive models, including Llama4-Scout (109B) and Qwen3-VL (30B).

Table 2: Significant gains across the board. Kimi-VL sees a +3.17% jump in complex MathVerse tasks.

Key Takeaways from the Results:

• It's not just OCR: Simply telling the model "do OCR first" didn't help. The distraction happens deep in the latent space.
• Generic Guidance Fails: You can't just boost any random experts. Boosting "Math" experts helps with math images, but using "General" experts actually hurts performance.
• Top-1 vs. Top-K: Higher-parameter models like Llama4 (Top-1 routing) are more sensitive; they require a "lighter touch" in intervention compared to Top-K models.

Conclusion: A New Frontier for MoE

This work shifts the focus of multimodal research. We've spent years trying to align images and text; now we know that even if they are aligned, the MoE Routing Mechanism can still be biased by the "look" of the data.

Future Outlook: The next generation of VLMs shouldn't just be bigger; they need to be "Routing-Robust." We need training objectives that punish the model for ignoring reasoning experts just because the input happened to be an image. Until then, inference-time steering like the one proposed here is our best tool to bridge the "Seeing vs. Thinking" gap.

Paper Source: Zhejiang University & Alibaba Group, 2026.