When Do Relative Policy Differences Transfer in On-Policy Distillation?

What a relative policy difference actually encodes

On-policy distillation (OPD) normally trains a target model on its own generated answers, using token-level guidance from a stronger teacher. That setup assumes the teacher is clearly better than the target. The paper studies a variant that instead transfers the difference between an expert policy and a reference policy, so the teacher does not need to be stronger than the target in absolute terms.

The paper's derivation makes this concrete. The log-probability ratio between a proxy expert and a proxy reference is treated as an implicit reward for KL-regularized policy optimization, a standard objective that trades task reward against staying close to a reference policy. Differentiating that objective yields an on-policy update in which the target keeps its own reference and reweights it by the relative change between expert and reference. Conventional OPD appears as the special case where the proxy expert is the teacher, the proxy reference is the target's reference, and the transfer strength ratio equals one.

The authors call the general form any-to-any policy transfer, because the proxy pair and the target are chosen independently. The derivation also separates two questions that earlier work tended to merge: whether a source pair encodes a useful improvement, and whether a particular target can realize it.

Task-trained differences transfer; scale gaps mostly do not

The central experiment fixes Qwen3-8B as the target and varies where the proxy difference comes from. Task-optimized pairs, built by RL or SFT on math and code data, all improved the target. The reported target gains ranged from 9.78 points for a Qwen3-1.7B-RL pair to 32.23 points for a Skywork-7B pair referenced against Qwen2.5-7B.

Scale-only pairs behaved differently. Four pairs built purely from larger versus smaller models produced much smaller gains, and some were negative: a Qwen3-8B expert against a Qwen3-4B reference gave a target gain of −1.025 points, and a Qwen3-14B expert against Qwen3-4B gave 0.018 points. These pairs still had positive proxy capability gaps, so the failure is not explained by the expert being no better than its reference.

The paper's interpretation is that KL-regularized task optimization makes the log-ratio encode a task-relevant direction, and this reading extends approximately to RL or offline SFT experts even without explicit KL regularization. A scale gap between independently trained models need not point in any task-relevant direction. The practical implication is that a bigger expert is not automatically a better supervision source; what matters is whether the difference was produced by training on the task.

The expert supplies capability, the reference isolates the residual

The two models in the proxy pair do different jobs. The expert carries the capability to be transferred, while the reference removes behavior the two models share and leaves the residual difference that the target is asked to learn. The paper tests this by holding the expert fixed and swapping references.

With Qwen3-4B-RL as the expert, changing the reference changed outcomes substantially. Against a Qwen3-4B reference the target gained 26.84 points; against a Qwen3-1.7B reference it gained 25.85 points; against a Qwen2.5-3B reference it gained 25.85 points in the reported table. The reference need not be the expert's original training base, but its choice shapes what residual reaches the target.

The reference also determines whether the difference arrives in a form the target can use. In one intervention, replacing the Qwen3 reference with a Qwen2.5 reference turned Qwen2.5 targets from near-zero or negative gains into positive ones, with gains of 6.74, 2.25 and 5.76 points, while Qwen3 targets stayed successful. The authors read this as evidence that transfer failure is not an intrinsic limit of the target or of the difference; the reference controls presentation.

A useful difference still needs a compatible target

Applying the same difference to different targets produced sharply different results. With a Qwen3-1.7B-RL proxy pair, Qwen3 targets gained between 9.78 and 29.09 points, while Qwen2.5 targets under a Qwen3 reference gained almost nothing, with target gains between −2.46 and 2.72 points. Whether a difference helps is therefore a property of the target as well as of the difference.

A natural guess is that targets close to the proxy pair benefit most, but the data reject that. Neither the performance realization ratio nor transferred accuracy was monotone in the base KL between proxy and target reference. Qwen3 targets still gained at the largest measured base KL, around 0.65 realization when base KL exceeded 1.2. The split followed model family rather than policy distance.

The paper states the requirement directly: the induced update must move the target toward the expert and must generalize beyond the sampled rollouts. Raw policy distance did not predict transferability, whereas an appropriate reference rescued otherwise unsuccessful transfer. That makes reference selection a lever, not a formality.

How the study measured transfer and its limits

All experiments optimized the same objective with a fixed transfer strength ratio of one, using models from the Qwen3 and Qwen2.5 families. For efficiency, training used the top-16 token distribution rather than the full vocabulary. Proxy pairs were produced by RL or SFT on DeepMath-103K for math and Eurus-RL-Code for code with a rule-verifiable reward.

Transfer was measured in performance space. The proxy gain is the expert's average task performance minus the reference's, the target gain is the optimized target's performance minus its reference's, and the realization ratio divides target gain by proxy gain. Evaluation covered eight math benchmarks with 16 samples each and three code benchmarks with 8 samples each, at temperature 1.0 and top-p 1.0.

The paper also reports KL-based diagnostics that describe distributional change rather than accuracy, including the proxy expert–reference difference, the proxy–target reference gap, and the difference between the target's change and the change induced by the proxy log-policy difference. These are distinct quantities from the performance gains and should not be read as substitutes for them. The top-16 truncation and the two model families bound how far the results generalize.

Choose the reference before the bigger expert

The paper's most decision-relevant comparison is between two ways of making a proxy pair look strong. Increasing the expert's scale produced positive proxy gains but little or negative target gain, while task-specific training produced differences that transferred consistently. If the goal is reusable supervision, the evidence favors a task-trained expert over a merely larger one.

The second priority is the reference. It filters shared behavior, sets the residual the target sees, and in the reported intervention changed Qwen2.5 targets from failure to positive gains. Raw policy distance between proxy and target did not predict success, so selecting a reference by closeness alone is not supported by these results.

The remaining condition is the target itself: the induced update must move it toward the expert and generalize beyond sampled rollouts. The paper's evidence for that condition is a family-level split in its own experiments, not a general rule for arbitrary model pairs, and the fixed transfer strength and top-16 token truncation limit how far the numbers should be extended.

Sources used in this answer