How does multi-evidence self-distillation improve VideoLLM reinforcement learning?

Sequence rewards leave token-level credit unguided

Reinforcement learning with verifiable rewards gives VideoLLMs reliable outcome supervision, but the paper argues that sequence-level rewards offer limited token-level guidance. A final-answer reward cannot reliably indicate whether intermediate steps correctly used temporal and spatial evidence, so the model may be reinforced for an answer without learning which reasoning tokens deserved credit.

On-policy self-distillation addresses this by conditioning a self-teacher on privileged information to provide dense token-level supervision on student-generated trajectories. The paper identifies two remaining problems: aggregating heterogeneous evidence inside a single teacher context obscures cross-evidence agreement and conflict, and it is unclear whether teacher guidance should refine reward-based updates or instead correct failed trajectories.

Three teachers share parameters but see different evidence

MeSD constructs three evidence-conditioned teachers with shared parameters. The Answer Teacher sees only the semantic ground-truth answer, excluding temporal intervals, keyframe timestamps, and bounding box coordinates. The Answer-Temporal Teacher adds annotated time intervals or keyframe timestamps, and the Answer-Spatial Teacher adds object names and transformed bounding boxes. All three evaluate the same student-generated prefixes without generating additional rollouts, and their predictions are detached to prevent gradient backpropagation.

This design is what makes cross-evidence comparison possible. Because the answer context is held constant while temporal or spatial evidence is added separately, the paper can measure evidence-specific preference shifts relative to the Answer Teacher rather than mixing all privileged information into one context.

Fusion separates consensus from evidence-specific residuals

For each candidate token, MeSD defines the preference change induced by temporal or spatial evidence as the log-probability difference from the Answer Teacher. A positive value means the added evidence increases support for that token relative to the anchor; a negative value means support decreases.

The fused distribution combines a teacher-common preference with a gated teacher-specific residual. The common term uses the geometric mean across non-anchor teachers, which emphasizes tokens supported by all of them, since low probability from any single teacher reduces joint support. The residual is the difference between the arithmetic mean and the geometric mean in log space, capturing complementary support from individual teachers. Two gates control the residual: an agreement gate reduces its contribution when evidence-specific preferences have opposing signs, and a magnitude gate limits it relative to the common preference. The gated residual is added back and used to reweight and normalize the Answer Teacher distribution.

When semantic answers are unavailable, the method falls back to an unprivileged anchor teacher, and retains that distribution if no auxiliary teacher is available.

Verification decides between refinement and correction

MeSD uses a task-specific verifier to classify each trajectory as Success, Failure, or Indeterminate. For Success and Indeterminate trajectories, token-level advantage refinement adjusts advantage magnitudes while preserving reward-derived signs. The refined token advantage substitutes into the GRPO clipped objective, with a strength parameter controlling teacher reweighting and a clipping range bounding the token-specific weight.

For verified failure trajectories that contain the required evidence, MeSD instead applies failure-conditioned distillation. This replaces the GRPO policy loss with a reverse-KL objective targeting the fused teacher distribution, conditioned on the same student-generated prefixes. The teacher target is detached, so gradients flow only through the student distribution. Crucially, this loss does not depend on the advantage, so it can supply corrective supervision even when group-relative advantages vanish.

The unified objective mixes the two branches per trajectory, with a weight on the distillation loss. Teacher targets stay detached throughout training, the shared teacher parameters are updated by exponential moving average after each student update, and the teacher strength is gradually decayed before setting the refinement and correction switches to zero to recover standard GRPO.

Gains hold across benchmark families and model sizes

The paper reports consistent gains over reinforcement learning and self-distillation baselines on multiple video benchmarks. Evaluation covers joint spatiotemporal reasoning on V-STaR, which has two reasoning chains (what–when–where and what–where–when) and lets the authors separate answer accuracy from temporal and spatial grounding; temporal grounding on TVGBench and Charades-STA; and general or long-video understanding on WorldSense, VideoMMMU, LongVideo-Reason-eval, and Video-MME-v2.

The discussion states that improvements extend across Qwen3-VL model sizes, and that ablations support evidence separation, gated fusion, and selective correction. The training data is the STGR-RL set from Open-o3-Video, totaling 37,231 samples: 32,231 video-based and 5,000 image-based spatial grounding samples. MeSD reuses the existing annotations without adding human-written or model-generated ones, reorganizing them into separate evidence-conditioned contexts.

What the annotations allow and what they do not

The evidence-conditioned teachers depend on annotation availability. General video QA provides semantic answers, temporal QA supplies time intervals, and joint spatio-temporal QA includes keyframe timestamps and object-level bounding boxes. For pure temporal QA, the answer is itself a time interval, so it contributes temporal evidence rather than a separate semantic answer. This means the three-teacher structure is only fully populated where the corresponding annotations exist.

The paper also bounds the failure-correction claim. The reverse-KL argument concerns the distribution-level objective: for eligible verified failures, distribution mismatch can supply a correction signal even when credit refinement provides none. The paper explicitly notes that this does not guarantee a correct answer or a nonzero aggregate gradient with respect to the model parameters. Failures excluded by the strict eligibility rules keep credit refinement instead.

The priority is selective correction, not blanket distillation

The most decision-relevant distinction in this paper is not that teacher signals help, but when they should act. MeSD preserves reward-derived advantage signs on Success and Indeterminate trajectories and reserves reverse-KL correction for verified failures that contain the required evidence. That split is the paper's central claim about how to combine outcome rewards with dense token supervision.

For readers comparing post-training recipes, the practical threshold is therefore verification outcome plus evidence availability: a failed trajectory without the required evidence is not corrected by distillation, and a successful trajectory is refined rather than overwritten. The paper's own framing is that this selective use of the fused teacher distribution, built from existing annotations, is what separates MeSD from methods that reweight tokens uniformly across successful and failed trajectories.

Sources used in this answer