[CVPR 2026] Demystifying OPD: Solving the "Repetition Collapse" in LLM Training

TL;DR

On-Policy Distillation (OPD) is theoretical gold—it trains a student on its own distribution to avoid the "distribution shift" seen in offline KD. However, it often fails in practice due to Abrupt Truncation-Repetition Inflation. This paper identifies the mathematical root of this collapse and introduces Stable-OPD, a framework that uses Mixture Distillation and KL-regularization to boost math reasoning performance by up to 7.2%.

The Problem: The "Repetition Saturation" Loop

In standard OPD, the student generates a response, and a teacher model provides token-level rewards based on the reverse-KL divergence. The authors discovered a lethal pathology:

1. Implicit Bias: Repetitive tokens (e.g., repeating the same reasoning step) often receive disproportionately large advantages from the teacher model.
2. Self-Reinforcement: Because the student samples from its own policy, as it begins to repeat, it encounters more "repetitive states."
3. Positive Feedback: These repetitive states dominate the gradient updates, "hacking" the distillation objective until the model only produces long, truncated, and useless noise.

Figure 1: The sudden spike in rollout length and the corresponding drop in validation accuracy during standard OPD.

Methodology: Stable-OPD

To break this cycle, the authors propose a dual-anchor strategy:

1. Mixture Distillation

Instead of relying purely on the student's own (potentially degenerate) rollouts, the model is trained on a mixture of:

• On-policy student rollouts: To maintain the benefits of OPD.
• Off-policy "Golden" data: High-quality, human or strong-teacher-written solutions that are complete and non-repetitive.

This acts as a "distributional anchor," ensuring the gradient always sees what a "good" trajectory looks like.

2. KL-Regularized Constraints

The authors add a penalty for the student drifting too far from its original reference policy (${\pi }_{ref}$). This prevents the "policy drift" that occurs when the student starts chasing the high rewards found in repetitive sequences.

${\mathcal{L}}_{\mathrm{Stable}-\mathrm{OPD}}(heta)={\mathcal{L}}_{\mathrm{OPD}}(heta)+{\lambda }_{\mathrm{gold}}{\mathcal{L}}_{\mathrm{SFT}}(heta)+{\beta }_{\mathrm{KL}}\mathbb{E}[\mathrm{KL}({\pi }_{heta}\parallel {\pi }_{ref})]$

Figure 2: Empirical evidence showing that repetitive tokens systematically receive higher advantages in standard OPD, necessitating the Stable-OPD constraints.

Experiments & Results

Stable-OPD was tested on six mathematical benchmarks (AIME, MATH500, etc.) using Qwen2.5 backbones (1.5B and 7B).

• Stability: Unlike standard OPD, which collapses after ~30 steps, Stable-OPD remains stable throughout training.
• Performance: On the Qwen2.5-Math-7B backbone, Stable-OPD reached an average accuracy of 47.6%, outperforming competitive RLVR (Reinforcement Learning from Verifiable Rewards) methods like DeepSeek-style GRPO (45.5%) and PRIME-Zero (40.8%).
• Ablations: The authors found that Mixture Distillation is the primary driver of performance, while KL-regularization provides the necessary fine-grained stability.

Figure 3: Contrast between OPD (collapse) and Stable-OPD (stable) across truncation and repetition rates.

Critical Insight & Conclusion

The real takeaway here is that On-Policy training is a double-edged sword. While it fixes distribution mismatch, it introduces a "feedback loop" that can amplify the slightest bias in a reward signal or a teacher's log-probability.

Stable-OPD proves that we don't need complex, multi-stage RL pipelines to get SFT-beating performance; we just need a robust way to anchor the student's exploration. For anyone building reasoning models, this paper provides a clear blueprint for using distillation without the fear of training collapse.