Does transferring teacher-relative shifts improve multi-teacher on-policy distillation?

What Δ-MOPD transfers instead of the endpoint

Multi-teacher on-policy distillation (MOPD) trains a student on its own rollouts, using scores from several frozen teachers. The usual recipe transfers each teacher's endpoint policy: the final distribution the teacher produces after post-training. The paper argues that this endpoint bundles two different things — what post-training actually changed, and preferences the teacher inherited from its own base model. When a teacher's base differs from the student's starting checkpoint, endpoint supervision carries that inherited difference along too.

Δ-MOPD changes the transferred object. For each teacher it computes the logit shift between the teacher and that teacher's own base, then re-anchors the sum of those shifts at the student's frozen initialization. Writing z for logits at a student-generated state, the endpoint target adds each teacher-minus-student term to the student's logits, while the shift target adds each teacher-minus-its-base term. When a teacher's base equals the student's initialization, the two constructions produce the same term, so only cross-origin teachers are affected.

The paper separates two decisions that MOPD recipes usually conflate: teacher selection, meaning which teachers supervise a state, and target construction, meaning how their scores define the distribution to match. Every comparison holds teacher selection fixed and changes only how the cross-origin teacher Polaris is represented, so the contrast is attributable to target construction rather than to a different teacher set.

Inherited base pull can outweigh the post-training shift

The paper decomposes the endpoint reward of a teacher into a shift term, the teacher-minus-base logit difference, plus a base-reference pull, the base-minus-student difference. At initialization that pull is exactly the difference inherited from the teacher's base. The authors' diagnosis is that this inherited pull can be larger than the post-training change the teacher was supposed to contribute, so endpoint supervision spends gradient on reproducing the base rather than the specialization.

The diagnostics support that reading. In the two-teacher mechanism run, the endpoint composite's target-to-student forward KL averages 0.1522 over the late-training window (steps 300 to 380), while Δ-MOPD's averages 0.0307 — roughly five times closer to the student. The endpoint composite's aggregate teacher-term norm is 10231.5 against 3247.5 for Δ-MOPD, and its cancellation value is 0.1363 versus 0.2871, where 0.2929 is the reference for two orthogonal equal-norm vectors. The paper reads the higher cancellation as more balanced composition across teachers.

These are diagnostics of the learning signal, not benchmark scores. The KL figure is a forward KL from target to student computed over the full 151,665-token vocabulary, and the paper states it measures target tracking rather than the training objective. The norm, cosine, cancellation, and sign-conflict statistics are computed at student-visited prefixes, so they depend on the rollout protocol and pooling rule and are meant to be read within a run.

Gains appear when three teachers score one state

In common-domain composition, several teachers score the same rollout from one prompt domain and their signals merge into a single target. With three composed teachers, Δ-MOPD exceeds endpoint composition by 4.11 Math points and 1.95 five-benchmark points. With two teachers, the abstract reports that it matches endpoint accuracy, and the mechanism run shows the split: Δ-MOPD is 3.40 points lower on MATH-500 but 1.26 points higher on GPQA-Diamond, leaving the English Math macro 1.03 points lower and the Science/IF macro 0.58 points higher.

The scaling run adds JustRL to Nemotron and Polaris. The third shift enters at a magnitude comparable to the other two (norms 12,891, 11,114, and 6,490), and two of its pairwise cosines stay near zero (0.072 and 0.062) while the third rises to 0.315. Across the recorded two- and three-teacher diagnostics, shift composition lowers target displacement, reward variance, and gradient magnitude; at three teachers its final target-to-anchor KL is 0.483 rather than 0.695. The cost is 11.0% more per-update time (167.1 versus 150.6 seconds) because the Polaris precursor must also be scored.

The paper's interpretation is that removing inherited base differences yields a target closer to the student and, under composition, more balanced across teachers. That is the authors' mechanism claim; the benchmark comparison and the geometry statistics are the evidence offered for it.

Routing helps only when teacher signals accumulate

In routed-domain distillation, prompts from different domains go to the corresponding specialist, so each state is supervised by one teacher. The paper tests two schedules. Under interleaved routing, where each update involves one teacher and domains are mixed from the first update, the two targets perform comparably. Under phased routing, where one domain phase follows another and actor and optimizer state carry across the switch, Δ-MOPD achieves higher mean performance in both phase orders and reduces the observed order gap from 10.50 to 6.42 points.

The phased numbers show where the difference sits. In the Science/IF-to-Math order, Δ-MOPD reaches a 34.39 Math macro and 22.89 Sci/IF macro against 28.02 and 18.25 for endpoint. In the Math-to-Science/IF order, it reaches 42.20 and 27.23 against 40.87 and 25.23. Because the Science/IF phase is identical across methods, the contrast lies in the Polaris Math phase.

The paper is careful about the strength of this evidence. Same-order comparisons are exact paired McNemar tests on identical items, and the only comparison below p = 0.05 is MATH-500 in the Math-to-Science/IF order (56 versus 36 discordant items, p = 0.047), which does not survive correction across the ten benchmark-by-order comparisons. The authors therefore describe the phased results as supporting evidence that the benefit may extend to signals accumulated across training phases, not as a confirmed effect.

Fewer updates, but each update costs more

Δ-MOPD needs each teacher's precursor checkpoint, so it scores one more model per update. In the single-teacher acquisition run, it is 27.3% slower per update; at equal update count it costs 27.3% more in the one-teacher setting and 12.0% more with two teachers, below a naive doubling because rollout generation, optimization, batching, and parallel scoring are shared.

The compute saving comes from reaching a target score in fewer updates. Δ-MOPD first exceeds endpoint's best evaluated English-Math macro at step 100, while endpoint attains that best score at step 195. With mean step times of about 252 and 198 seconds on four allocated H100s, allocated costs are 28.0 versus 42.9 H100-hours, a 34.7% reduction reported as 35%. Under the paper's plateau criterion, Δ-MOPD stabilizes near step 75 and endpoint near step 125, corresponding to 21.0 versus 27.5 H100-hours.

These figures measure allocated device time, not FLOPs, and do not correct for utilization. The acquisition run trains only on BigMath prompts, and its English-Math macro across checkpoints is 47.90/45.65/47.68 for Δ-MOPD against 45.03/46.18/46.54 for endpoint at steps 100/150/195. On GPQA-Diamond and IF-Eval the ordering flips with decoding mode: at step 100, Δ-MOPD leads by 0.91 points on the Avg@4 macro while greedy decoding favors endpoint by 1.07 points.

A projected fourth teacher lifts Math only

The paper extends the construction to a second cross-origin teacher, Qwen3-DAPO-449, which is tokenizer-identical to its own Qwen3-8B precursor but not fully special-token compatible with the DeepSeek/Qwen2 anchor. The authors compute the Qwen3 shift inside the Qwen3 family and project only overlapping token coordinates onto the anchor vocabulary, verifying 151,660 token-string overlaps and 151,658 shared tokens with identical ids.

Adding this projected shift raises the Math macro from 33.50 to 36.82 and the five-suite macro from 26.19 to 28.04. All three Math benchmarks improve, with the largest gain on AIME 2025, while Science/IF does not improve. Because cross-tokenizer endpoint logits are not placed on the anchor vocabulary, no paired endpoint control is defined for this extension, so it demonstrates that the construction extends rather than that it beats endpoint supervision at four teachers.

Choose shift targets by how teachers meet at a state

The decision rule the evidence supports is about the supervision pattern, not the teacher count alone. When several teachers score the same state and their signals are merged into one target, shift targets are the better transfer object: three composed teachers give the clearest margin, and two composed teachers give parity rather than a loss. When each update involves exactly one teacher, as in interleaved routing, the two targets are comparable, so the extra precursor scoring buys little. Phased routing sits in between and favors shift targets in both orders, but with statistical support the authors themselves describe as limited.

The requirements matter for that choice. Δ-MOPD needs each teacher's precursor and a compatible tokenizer or an explicit projection rule, unlike black-box on-policy distillation, and the evaluation covers public reasoning specialists with common-domain composition on Math prompts and Science/IF measured through held-out evaluation. Within those bounds, the paper's priority is that target construction is an independent design axis in MOPD, complementary to teacher selection: a practitioner who has already chosen teachers can still change what is transferred, and the evidence says that change pays off specifically where teacher signals are combined at a state.

Sources used in this answer