Reading, Not Thinking: Breaking the Modality Barrier in Multimodal LLMs

TL;DR

Why do state-of-the-art Multimodal Large Language Models (MLLMs) fail at grade-school math when the question is a screenshot rather than text? This paper identifies that the "modality gap" is not a lack of intelligence, but a failure of perception and reasoning triggers. By using a novel Self-Distillation method—training a model to mimic its own text-based logic when looking at pixels—the authors achieved a staggering recovery in performance, such as boosting GSM8K image-mode accuracy from 30% to 92%.

The "Modality Gap" Mystery

We have long assumed that if a model can "see," it can "read" and then "think." However, empirical evidence shows a massive performance drop when the exact same semantic content is moved from a text prompt to an image. This gap is most brutal in structured reasoning tasks like math (GSM8K) and coding (HumanEval).

The authors' first major insight is that this gap is distribution-dependent. Interestingly, models often perform better on natural document images (like arXiv PDFs) than on clean, synthetic renderings. This suggests that the noise and layout of real-world documents are better represented in pretraining than the "perfect" synthetic images we use for testing.

Methodology: Diagnosing the Failure

To understand the "Why," the researchers utilized a grounded-theory error analysis, classifying over 4,000 errors.

1. Reading vs. Thinking

The taxonomy revealed that image-mode input selectively targets reading-related failures. Calculation errors (misreading a digit) and format errors (misreading an instruction) increased by 50%. Meanwhile, conceptual knowledge and abstract reasoning remained stable. The verdict: The model's "brain" is fine; its "eyes" are deceiving it.

2. The Chain-of-Thought Collapse

Perhaps the most startling discovery was the reasoning collapse. In text mode, models typically generate long, step-by-step Chain-of-Thought (CoT) responses. In image mode, they "lazy out," producing 19x shorter responses and jumping straight to (usually wrong) answers. The visual modality fails to trigger the model's sequential reasoning engine.

The Solution: Self-Distillation

If the model knows how to solve the problem in text mode, why not use that knowledge to train its vision mode?

The authors proposed Self-Distillation:

1. Generate high-quality CoT traces using the model in Text Mode.
2. Pair these text-based "gold" reasoning steps with the Image Mode version of the input.
3. Fine-tune the model (Language Model and/or Vision Encoder) using LoRA to ensure the image input produces the same high-quality CoT.

Experimental Results

This approach virtually eliminated the gap. On GSM8K, the baseline gap of ~63% was slashed to a mere 1.37%.

Crucially, adapting the Language Model (LM) was found to be more vital than adapting the Vision Encoder (ViT). This suggests that the primary issue is how the LM interprets visual tokens, rather than the tokens being inherently "unreadable."

Critical Insights & Future Outlook

• Rendering Matters: Font choice (e.g., handwriting) can swing accuracy by up to 47%. Benchmarks that don't specify rendering parameters are scientifically incomplete.
• Efficiency: Distillation on a single benchmark (GSM8K) transferred to others (MMLU, ARC) without catastrophic forgetting, suggesting a general "alignment" of visual perception to linguistic logic was achieved.
• Limitations: While this fixes "reading," it doesn't necessarily address complex spatial reasoning where the arrangement is the logic (like geometric diagrams), which remains a frontier for MLLMs.

In conclusion, the modality gap is a bridgeable chasm. By acknowledging that models are "Reading, Not Thinking" when they fail in vision-text tasks, we can stop building bigger models and start building better-aligned ones.