[ICLR 2025] MLLMs as Classifiers: Are They Really Worse Than ResNet?

TL;DR

Recent studies suggested that Multimodal Large Language Models (MLLMs) lag behind traditional supervised models in simple image classification. This paper reveals a different reality: the "gap" is mostly a result of noisy ImageNet labels and rigid evaluation protocols. By introducing a cleaned multi-label ground truth (ReGT) and a semantic mapping layer (CW+), the authors show that models like GPT-4o are much closer to SOTA than previously thought.

The Motivation: Stop Blaming the Model for Noisy Data

For years, ImageNet-1k has been the "gold standard," but it is notoriously dirty. It forces a single-label constraint on a multi-label world and contains roughly 20% labeling errors. MLLMs, which are trained on vast, diverse internet data, often "hallucinate" labels that are actually visually present but not in the ImageNet label set.

The authors argue that we have been using the wrong "yardstick" to measure these models. They identify three major protocol flaws:

1. The OOP Penalty: Models are marked wrong for saying "laptop" if the class is "notebook computer."
2. Multiple-Choice Inflation: Random distractors make the task too easy, overestimating MLLM "intelligence."
3. Ground Truth Noise: Models are penalized for being more accurate than the original (often wrong) annotators.

Methodology: Solving the Mapping Problem

The core technical contribution is the CW+ (Closed-World Plus) protocol. Instead of forcing the model to strictly generate a string (which is prone to lexical failure), the authors allow the MLLM to speak freely and then use a text encoder to map that response to the nearest class in the embedding space.

Figure 1: Comparison of Open-World (OW), Multiple-Choice (MC), and the proposed Closed-World (CW+) setups.

By using state-of-the-art encoders like SigLIP 2 or Qwen3-Embedding, the authors remove the requirement for "exact string matching," which accounted for a significant portion of "perceived" errors in prior work.

Experimental Results: The Gap is Closing

When evaluated on the new ReGT labels (625 classes), MLLMs saw a massive performance boost.

• PaliGemma 2: +10.8% Accuracy gain.
• GPT-4o: +6.0% reaching 82.36% in CW+.
• Contrast: Supervised models like EVA-02 only gained 1.4% (or even lost performance), suggesting they are overfitted to the "noise" of the original ImageNet.

Figure 2: Comprehensive results showing the delta between original labels (ImGT) and reannotated labels (ReGT).

MLLMs as "Super-Annotators"

In a fascinating case study, the authors used GPT-4o to flag errors in human annotations. In 50% of difficult cases, the human annotators realized the MLLM was correct and the initial ground truth was wrong. This suggests a paradigm shift: MLLMs shouldn't just be evaluated by benchmarks; they should be used to build them.

Critical Analysis & Takeaways

The Good: This work provides the first "fair" comparison between generative MLLMs and discriminative ViTs. It proves that embedding-based mapping is a lightweight, effective alternative to slow, constrained decoding.

The Limitations: The reannotation still only covers 625 of the 1,000 classes. Fine-grained biological categories (like specific bird species) remain a challenge for both MLLMs and non-expert human annotators.

Future Outlook: We are moving toward a "Post-ImageNet" era where "Zero-shot" performance in the Open-World is more valuable than benchmark-specific top-1 accuracy. As MLLMs continue to scale, the bottleneck is no longer the model architecture, but the quality of our ground truth.

Note: This blog post is based on the paper "Multimodal Large Language Models as Image Classifiers" from the Visual Recognition Group at CTU in Prague.