[CVPR 2025] Unified GRPO: Unlocking Seamless Interleaved Text-Image Generation

TL;DR

While unified models like VILA-U and Show-o have mastered multimodal understanding, they still struggle to "think and draw" in a continuous stream. This paper proposes a breakthrough post-training strategy: a Warm-up phase followed by Multimodal Group Relative Policy Optimization (GRPO). By treating text and images as a single decision-making trajectory and applying step-wise rewards, the model achieves SOTA performance in visual storytelling and complex reasoning.

Background & Motivation: The Modality Switch Gap

Current Large Multimodal Models (LMMs) are often "binary"—they either answer a question in text or generate an image from a prompt. However, human-like reasoning often requires interleaved outputs (e.g., "First, do this [Image], then follow with this [Text]").

The core bottleneck isn't the model architecture, but the data and the optimization objective. High-quality interleaved data is scarce, and standard Supervised Fine-Tuning (SFT) often leads to "catastrophic forgetting" of the model's pre-trained understanding capabilities.

Methodology: The Multimodal Policy Shift

The researchers propose a two-step solution to activate the latent interleaved capabilities of unified models:

1. The Hybrid Warm-up Stage

Instead of jumping straight into RL, the authors use a 0.3M sample "warm-up" dataset (ActivityNet, GenHowTo, OpenStory++) mixed with 2M samples of standard understanding and T2I data. This ensures the model learns the pattern of interleaving without losing its basic skills.

2. Multimodal GRPO with Hybrid Rewards

The highlight of the paper is the extension of DeepSeek's GRPO to the multimodal domain. Unlike PPO, GRPO compares a group of generations to estimate advantages without a separate critic model.

• Unified Trajectory: The model treats $\{y_{text},y_{image},y_{text}\}$ as one sequence.
• Hybrid Rewards: The authors combine three reward types:
  • Textual ($r_t$): Relevance to the prompt.
  • Visual ($r_v$): Using ImageReward to assess quality and alignment.
  • Format ($r_f$): Penalizing the model if it fails to use <think> and <vis> tags correctly.

Figure 1: Overview of the reinforcement fine-tuning framework using hybrid rewards.

3. Process-Level Rewards

Sparse, end-of-sequence rewards are insufficient for long interleaved stories. The authors introduce Process-level rewards, providing feedback at every modality switch. This granular guidance helps the model realize "Aha! I should generate an image here that matches the previous paragraph."

Experimental Results: Setting a New Bar

The approach was tested on MMIE and InterleavedBench, the two most rigorous benchmarks for this task.

• MMIE Results: The model reached 59.50%, outperforming specialized models like MiniGPT-5 and Anole.
• InterleavedBench: Significant gains in "Text-Image Coherence" (TIC) and "Helpfulness," proving that the images generated weren't just pretty, but contextually accurate.

Table 1: Comparison of the proposed method against SOTA unified models on MMIE.

Qualitative Insights: Visual Storytelling

The model demonstrates an impressive ability to follow complex instructions. For example, when asked to "Continue a story about a team huddling," it generates text describing the intensity, followed by a visually consistent image of a stadium, and then continues the narrative flow.

Figure 2: Examples of coherent, interleaved text-image generation across different tasks.

Critical Analysis & Conclusion

Takeaway: This work proves that RLHF (and specifically GRPO) is not just for LLM reasoning; it is the "secret sauce" for multimodal alignment. By shifting the objective from "predicting the next token" to "maximizing multimodal coherence," the authors solved the modality transition problem.

Limitations:

1. Base Model Dependency: The method refines existing capabilities but doesn't fundamentally improve the underlying vision encoder's resolution.
2. Hallucination: Complex logical reasoning can still lead to visual hallucinations where the image content contradicts the text.

Future Outlook: The next frontier will likely involve Multimodal Chain-of-Thought (CoT), where the model uses internal "hidden" tokens to plan its visual-textual interleaved steps more deliberately.