Unmasking On-Policy Distillation: Why Teachers Sometimes Lead Students Astray

TL;DR

On-policy distillation (OPD) is the secret sauce behind today’s top-tier reasoning models, yet it is often applied blindly. A team from Apple has developed a diagnostic framework that peers into the "moment-to-moment" interaction between teachers and students. Their findings? Distillation is most effective when the student is already failing, and a teacher that is "too smart" might actually provide gradients that a smaller student simply cannot comprehend.

The Motivation: Moving Beyond Aggregate Metrics

In the current LLM landscape, we often judge distillation by the final benchmark score. If the student’s MMLU goes up, the distillation "worked." However, this ignores the micro-dynamics of reasoning. A teacher might be 95% helpful but 5% harmful—perhaps by forcing a specific stylistic choice that distracts the student from the actual logic.

The authors ask three critical questions:

1. Which teacher? (Self-distillation vs. External models)
2. Which context? (Raw traces vs. Summarized logic)
3. When does it help? (At which specific tokens?)

Methodology: The Gradient Alignment Score

To solve this, the researchers defined an Ideal Gradient—the theoretical update that would most improve a student's chance of getting the right answer at any given node in a reasoning tree.

1. The Ideal Reference

By sampling thousands of completions from a specific point (node), they calculate the empirical success probability ($P_{succ}$). The ideal gradient moves the model's logits toward paths that statistically lead to "Correct" more often.

2. Measuring Alignment

The Gradient Alignment Score is the cosine similarity between this "Oracle" path and the actual gradient produced by distillation (e.g., GKD or MiniLLM).

• Positive Score: The teacher is pushing the student toward success.
• Negative Score: The teacher is actively sabotaging the student.

Key Insight 1: Distillation Helps the "Failures"

The most striking discovery is that teacher guidance is significantly more aligned with the ideal gradient on incorrect paths.

When a student is already on the right track, the teacher’s signal often becomes "noisy," focusing on trivial stylistic disagreements (e.g., "therefore" vs "so"). However, when the student is failing, the teacher provides a strong, corrective signal that pushes the student back toward the correct reasoning manifold.

Key Insight 2: The Comprehensibility Bottleneck

Is a bigger teacher always better? No.

• For a 0.6B Student: Self-distillation (using its own "correct" thoughts as context) provided a gradient 2-3x more aligned than a 14B teacher.
• For a 1.7B Student: The capacity gap was small enough that it could actually "understand" and benefit from the 14B teacher's superior knowledge.

The authors hypothesize that a student can only learn from a gradient it can "parse." If the teacher's reasoning style is too foreign or complex, the resulting gradient is effectively white noise to the student.

Methodology Detail: Targeted Rollouts

Calculating these scores for long-chain reasoning (like AIME math problems) is computationally expensive. The authors utilized a targeted-rollout algorithm. Instead of random sampling, they strategically allocated their compute budget to "branching points" where the teacher and student disagreed most sharply, using exponential depth windows to handle traces up to 30,000 tokens long.

Critical Analysis & Future Outlook

The paper shatters the idea of a "one-size-fits-all" distillation pipeline. On simple tasks (BoolQ), "wrong" demonstrations in the context only added noise. But on hard math (AIME), seeing a mistake provided a vital "contrastive" signal that helped the model avoid common pitfalls.

Takeaways for the Industry:

• Adaptive Distillation: We should gate distillation gradients based on student-teacher divergence.
• Rollout Weighting: Focus compute on "incorrect" trajectories where the alignment is highest.
• Teacher Selection: Don't just pick the biggest model; pick the one whose "thinking style" matches the student's capacity.

This diagnostic framework turns the "black box" of post-training into a measurable, engineering-led process.