[CVPR 2025] Granulon: Awakening Pixel-Level Encoders for Unified Multi-Granularity Reasoning

TL;DR

Granulon shifts the MLLM paradigm away from the "CLIP-dependency" by unlocking the potential of DINOv3, a pixel-level encoder. By introducing a text-conditioned controller and an adaptive token aggregation (AdaTA) module, it enables a single visual backbone to reason across multiple scales—from tiny textures to global scenes. The result? A 30% boost in accuracy and a 20% drop in hallucinations compared to traditional baselines.

The Granularity Dilemma: CLIP vs. DINO

For years, the MLLM community has relied on CLIP. CLIP is great at understanding "what" is in an image (global semantics) but terrible at "where" or "how many" (fine-grained details). On the other end of the spectrum, self-supervised models like DINOv3 are masters of local geometry and pixel-level textures but often "miss the forest for the trees," lacking the high-level semantic tokens that LLMs crave for reasoning.

The authors of Granulon identify that the missing link isn't a better encoder, but a dynamic bridge between these two worlds.

Methodology: How Granulon "Awakens" Pixels

Granulon introduces a unified "pixel-to-fine-to-coarse" reasoning pipeline. Instead of a static projection, it uses the input text to decide how much visual detail is needed.

1. Text-conditioned Granularity Controller

Before the image tokens reach the LLM, a controller looks at the question. If the question is "What color is the dog's ear?", it triggers a fine-grained mode. If it’s "What is the overall atmosphere?", it switches to coarse-grained. This prevents the LLM from being overwhelmed by irrelevant pixel noise or starving for detail.

2. Adaptive Token Aggregation (AdaTA)

The AdaTA module is the engine of the system. It processes features through three stages:

• Granularity-guided Pooling: Adjusts spatial resolution.
• Feature Clustering: Uses a mini-k-means approach to group tokens that share similar attention patterns.
• Feature Refinement: Selects the highest quality "semantic prototypes" to represent the image.

Figure 1: The Granulon architecture showcasing the AdaTA module and the Controller workflow.

Experimental Mastery: Crushing the Baselines

Granulon was tested across 5 benchmarks, including VQA, Image Captioning, and specialized Medical reasoning.

• SOTA Achievements: On SEED-Bench, Granulon (Qwen2.5 backbone) hit 58.8% Recall, significantly higher than CLIP (50.9%) and DINOv2 (41.4%).
• Hallucination Resistance: Perhaps the most impressive result is the reduction in "hallucination cascade." CLIP models often invent details that look semantically plausible but aren't in the image. Granulon’s pixel-level grounding keeps the LLM "honest."

Figure 2: Hallucination rate comparison across different visual encoders.

Deep Insight: Layer-wise Alignment

Why does this work? The authors performed a "Layer-wise Alignment Analysis." They found that CLIP-based models provide a good starting point but "plateau" as the LLM goes deeper into reasoning tiers. In contrast, Granulon’s cosine similarity to the ideal state continues to climb across the LLM layers, reaching a similarity of ~0.80 compared to CLIP's ~0.60. This proves that adaptive granularity provides a "scaffold" that supports the hierarchical nature of language reasoning.

Figure 3: Internal state alignment: Granulon vs. CLIP across LLM layers.

Conclusion & Future Outlook

Granulon demonstrates that the future of MLLMs may lie in dynamic visual perception. By moving away from fixed-resolution global encoders and adopting task-adaptive granularity, we can build models that are both more accurate and more trustworthy.

Future Work: The principles of Granulon could potentially be extended to video (temporal granularity) or 3D volumes (spatial-depth granularity), making it a foundational concept for next-generation embodied AI.