[CVPR 2026] Wallaroo: Unifying Multi-Modal Understanding, Generation, and Editing via Vanilla Next-Token Prediction

TL;DR

Wallaroo is an elegant, autoregressive baseline that proves "next-token prediction is all you need" for more than just text. By building upon Qwen2.5 VL and introducing a decoupled visual pathway, it seamlessly handles Understanding (VQA), Generation (Text-to-Image), and Editing (Image-to-Image) within a single 7B parameter transformer. It notably adds bilingual support (English/Chinese) and multi-resolution output flexibility.

Background & Positioning

The quest for Artificial General Intelligence (AGI) requires models that can both "see" and "create." Historically, the field split into two camps: Autoregressive (AR) models for understanding and Diffusion models for generation. Recent trends have tried to bridge this (e.g., using LLMs as encoders for Diffusion), but Wallaroo stays pure. It treats images as sequences of discrete tokens, placing it in the same lineage as Chameleon, Emu3, and Janus, but with a more streamlined approach to editing and resolution.

The Core Problem: The Representation Conflict

Why hasn't a single model mastered everything yet? The authors pinpoint a fundamental "Task Discrepancy":

1. Understanding requires high-level, semantic features (often compressed).
2. Generation requires low-level, reconstruction-heavy features.
3. Editing requires both—the semantics of the instruction and the structural integrity of the original pixels.

Prior works often forced one encoder to do everything, leading to "feature interference" where the model's performance in one task degraded the other.

Methodology: Decoupling and Staging

Wallaroo's architecture is built on a "minimalist principle."

1. Decoupled Architecture

Instead of a single "do-it-all" visual pipeline, Wallaroo uses:

• NaViT (from Qwen2.5 VL): For high-level understanding.
• VQ Tokenizer (LlamaGen): For discrete image generation.
• A Dual-Path for Editing: This is the secret sauce. For editing, the model takes both NaViT features (semantics) and VQ features (low-level details).

2. Four-Stage Training Strategy

To balance these capabilities without catastrophic forgetting, the authors devised a 4-step curriculum:

1. Preliminary Alignment: Training only the generation adaptors.
2. Joint Pretraining: Mixing understanding and generation data.
3. Scaling & Resolution: Training on 512x512 with <hw_info> tokens to handle various aspect ratios.
4. Unified Fine-tuning: Activating the "Editing" capability using high-quality instruction datasets.

Experiments: How Well Does it Perform?

Wallaroo was tested against massive baselines like Janus-Pro, OmniGen2, and Show-o2.

Understanding & Generation

In understanding tasks (MMBench), it reached 83.0, matching its ancestor Qwen2.5 VL. This proves that adding generation doesn't necessarily have to "break" the brain of the model. In generation (GenEval), it achieved a score of 0.75, which is competitive with other AR models like Janus-Pro, though slightly behind pure Diffusion models (which don't suffer from VQ quantization loss).

Image Editing

This is where Wallaroo shines. By using a 60% mask ratio during training to prevent simple "copy-pasting," the model learned to follow complex instructions. It achieved an overall editing score of 2.92, outperforming specialized editing models like AnyEdit.

Deep Insights: The "Editing" Bridge

One of the most profound discussions in the paper is the role of Image Editing. The authors speculate that editing serves as the "intermediate medium" bridging high-level understanding and low-level generation. They found that:

• Positional Encoding matters: Using 2-D encoding for understanding and 1-D for editing helps the model distinguish between "reading" and "modifying."
• Token Order is crucial: Putting low-level representations before high-level ones in the sequence significantly improved editing quality.

Limitations & Future

The primary "bottleneck" is the VQ Tokenizer. Because discrete tokens inherently lose some pixel-level detail, the output isn't always as "crisp" as a Diffusion model. Furthermore, the model currently uses three separate "heads" (Text, Gen, Edit), requiring users to switch modes manually—a hurdle for true "seamless" intelligence.

Conclusion

Wallaroo provides a strong, simple blueprint for the future of multi-modal AR models. It proves that with clever decoupling and a staged training curriculum, a single transformer can indeed be a Jack-of-all-trades without becoming a master of none.