Why looped models are hard to post-train
A looped language model (LoopLM) reuses the same Transformer blocks across several recurrent computation steps before emitting each token. That raises computational depth without raising parameter count, which is the appeal. The difficulty comes afterward: how to supervise the intermediate steps, not just the final output.
The paper lays out three existing routes and their costs. Reinforcement learning with verifiable rewards, including GRPO, assigns reward credit only to the terminal loop, so intermediate latent computation gets no direct reward supervision. Reward Latent Thought Trajectories extends credit across all loops but must retain per-loop log-probabilities, so memory grows linearly with the number of loops and long rollouts become expensive. On-policy distillation gives dense distributional supervision but needs an external, stronger teacher; for Ouro-2.6B-Thinking the authors state no stronger same-family LoopLM teacher is available. On-policy self-distillation builds a teacher from the same model using privileged information, but introduces a teacher-student context mismatch.
The gap the paper targets is therefore specific: dense supervision that reaches intermediate loops when no stronger external teacher exists. Its premise is that deeper loops receive more recurrent computation and so act as a stronger policy inside the same model.
The terminal loop becomes the teacher
LoopOPD treats the model's own terminal loop as a compute-privileged teacher and an intermediate loop as the student. The teacher is frozen, and supervision is applied on-policy: the student generates the rollouts, and the terminal loop's token distributions on those same rollouts serve as the training target. No external teacher and no privileged information are involved.
This is a distillation objective, not a reward signal, so the supervision is dense per token rather than sparse per episode. The teacher's advantage is not extra knowledge but extra recurrent computation, which is why the paper calls it compute-privileged.
The authors also characterize how a distillation update at one loop depth propagates to others, and derive sufficient conditions under which a single update yields simultaneous local improvement of the task surrogates at both the intermediate and terminal loops. Those conditions are an analysis result about the update, not a guarantee measured on benchmarks.
Refreshing the teacher turns distillation into self-improvement
A frozen teacher has an obvious ceiling: once the student catches up, the signal stops moving. Dynamic LoopOPD (D-LoopOPD) removes the freeze and refreshes the terminal loop teacher at each optimization step. Because teacher and student share parameters, student updates change the teacher too, closing a feedback loop the paper calls recurrent self-improvement.
The reported training dynamics are consistent with that mechanism. Measured at the student loop of depth 2, D-LoopOPD reached higher training reward and higher validation Avg@16 than frozen LoopOPD. The paper also reports that D-LoopOPD kept higher teacher and student token-distribution entropy than LoopOPD through most of training, with the gap narrowing later, which the authors read as less peaked distributions supporting more diverse rollouts and broader exploration.
That entropy reading is the authors' interpretation of a measured curve, not a separately tested causal claim.
Mathematical gains concentrate at the deeper model
The main comparison uses Ouro-Thinking models with four loops, evaluated on AIME 2024, AIME 2025, AIME 2026, OlympiadBench and BeyondAIME. Two metrics are reported for the AIME sets and must not be conflated: Avg@16 is the fraction of correct completions among 30 problems times 16 samples, i.e. 480 responses per benchmark; Pass@16 is the fraction of the 30 problems with at least one correct completion among 16.
On Ouro-2.6B-Thinking, D-LoopOPD reached Avg@16 of 71.67 on AIME 2024, 55.21 on AIME 2025 and 60.21 on AIME 2026, against vanilla values of 62.08, 48.96 and 55.42. Frozen LoopOPD landed in between at 65.21, 53.75 and 59.17. On OlympiadBench the ordering is tighter: 65.78 for D-LoopOPD, 64.88 for LoopOPD and 64.30 for vanilla. On BeyondAIME, D-LoopOPD scored 34.00 versus 35.00 for vanilla, so the gain is not uniform across every mathematical benchmark.
On the smaller Ouro-1.4B-Thinking the pattern repeats with smaller margins: D-LoopOPD reached 55.00 Avg@16 on AIME 2024 versus 51.25 for vanilla, and 47.92 versus 45.21 on AIME 2026. The paper's claim is that both methods beat existing baselines on average in mathematical reasoning, with D-LoopOPD further ahead of LoopOPD.
Math-only training also moves code and general reasoning
Post-training used only mathematical data, from a DAPO-Math-17k training split of 17,398 examples with integer ground-truth answers. The transfer results therefore test whether cross-loop supervision changes something broader than the training domain.
On Ouro-2.6B-Thinking, D-LoopOPD improved over vanilla on all five non-math benchmarks: MMLU 76.67 versus 76.09, MBPP 75.93 versus 74.87, MBPP+ 63.76 versus 62.43, HumanEval 79.27 versus 78.66, and HumanEval+ 75.00 versus 74.39. MMLU is accuracy over 14,042 questions in a 5-shot setting; the code benchmarks are Pass@1 with greedy decoding, 378 problems for MBPP and MBPP+, 164 for HumanEval and HumanEval+.
On Ouro-1.4B-Thinking, LoopOPD took the best MMLU result in its group at 68.05, while D-LoopOPD took the best MBPP, MBPP+ and HumanEval+ results. The margins are small, and the paper presents them as evidence that recurrent computation can serve as an effective supervision source, not as a claim of large cross-domain gains.
Loop 2 is the better student than loop 1 or loop 3
The student-depth ablation on Ouro-2.6B-Thinking holds the teacher at loop 4 and varies the student across loops 1, 2 and 3, evaluated by AIME 2024 Avg@16. The result is a trade-off rather than a monotone rule.
Loop 2 gave the best terminal performance, lifting loop 4 Avg@16 from 62.08 to 71.67, a 9.59-point gain, while its own Avg@16 rose from 39.16 to 47.50. Loop 3 started close to the initial terminal policy, 61.46 versus 62.08, leaving little room for distillation and a terminal gain of 5.21 points. Loop 1 had the largest gap to the teacher but very low standalone accuracy: its student Avg@16 stayed at 0.63 after training even though the terminal loop improved to 68.54.
The authors conclude that neither a larger teacher-student gap nor a stronger student alone determines effectiveness. In their setting, the intermediate loop at depth 2 balances student capability against teacher advantage.
Gains travel to deeper loops, not just the supervised one
Cross-loop evaluation on Ouro-2.6B-Thinking checks whether supervising one intermediate loop changes the others. Both LoopOPD and D-LoopOPD improved Avg@16 over vanilla at every depth, and D-LoopOPD had the highest Avg@16 among compared methods. At the directly supervised loop 2 it raised Avg@16 from 39.16 to 47.50, a gain of 8.34 points; loops 3 and 4 rose by 5.41 and 9.59 points to 66.87 and 71.67.
This is the paper's strongest support for its central claim: supervising intermediate computation improved both intermediate and terminal policies, so the extra recurrent depth was not wasted on the trained loop alone.
The scope is bounded by the evidence. All headline results come from Ouro-Thinking models with four loops trained on mathematical data, and the paper describes code and checkpoints as to be released upon acceptance, so the reported numbers are the authors' own evaluation rather than an independently reproduced benchmark. The sufficient conditions for simultaneous improvement are derived analytically and are not presented as measured guarantees on these benchmarks.
The teacher refresh, not the freeze, carries the gain
The decision-relevant comparison in this paper is not LoopOPD versus vanilla but LoopOPD versus D-LoopOPD. Freezing the terminal loop already delivers dense cross-loop supervision without an external teacher, and it improves mathematical reasoning. Refreshing that teacher as shared parameters move is what adds the further step: on Ouro-2.6B-Thinking, D-LoopOPD's AIME 2024 Avg@16 of 71.67 sits 6.46 points above frozen LoopOPD's 65.21 and 9.59 above vanilla.
The second thing to carry away is the student-depth choice. If a looped model is to be post-trained this way, the evidence favors an intermediate loop with real standalone capability but a clear gap to the terminal loop, which here was loop 2; a very weak student such as loop 1 left its own accuracy at 0.63 despite improving the terminal loop.
Read together, these two results set the paper's priority: the value of cross-loop distillation depends on keeping the teacher ahead of the student through refresh, and on picking a student loop where that gap is still usable.
Sources used in this answer
Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models
LoopOPD uses a frozen terminal loop as a compute-privileged teacher for an intermediate loop on student rollouts; D-LoopOPD refreshes that teacher each step and yields larger mathematical gains, with transfer to code and general reasoning.