[ArXiv 2026] V0.5: Bridging Generalist Priors and Sparse Rollouts for Stable LLM Reinforcement Learning

Executive Summary

TL;DR: V0.5 is a breakthrough in Reinforcement Learning with Verifiable Rewards (RLVR) that solves the "high variance vs. high compute" tradeoff. By using a frozen Generalist Value Model (V0) as a statistical anchor, it allows for stable training with extremely sparse rollouts (as few as 4 samples). It employs an Empirical Shrinkage Fusion and a Dynamic Budget Allocation mechanism to handle value model hallucinations, resulting in >10% accuracy gains on complex reasoning tasks like AIME and AMC.

Positioning: This work represents a shift from synchronous value learning (PPO) and brute-force sampling (GRPO) toward adaptive, prior-driven reinforcement learning. It is a "SOTA-shredding" refinement of the Generalist Value Model paradigm.

The "Coupling Dilemma" in LLM RL

Reinforcement learning for Large Language Models generally falls into two camps:

1. PPO-style: Uses a secondary "Critic" model. Problem: The Critic must be trained alongside the Actor, doubling VRAM requirements and trailing behind the non-stationary policy.
2. GRPO-style: Uses the mean of a group of rollouts as a baseline. Problem: To get an accurate (low-variance) mean, you need many rollouts (e.g., G=64). In long-reasoning tasks, this is computationally prohibitive.

If you try to use GRPO with "sparse" rollouts (e.g., G=4), the mathematical noise in the baseline estimation is amplified by the model’s massive parameter count, causing gradient explosion and "exploration collapse" (rapid entropy decay).

Methodology: The V0.5 Framework

V0.5 introduces a two-pronged statistical approach to "shrink" the error of the baseline.

1. Empirical Shrinkage Fusion

Instead of trusting only the rollout mean (${\bar{v}}_k$) or only the Generalist Prior ($V$), V0.5 uses a Shrinkage Estimator: ${\mu }^{\ast }=w\cdot {\bar{v}}_k+(1-w)\cdot V$ The authors prove (Theorem 3.3) that the optimal weight $w^{\ast }$ depends on the ratio between the prior's bias (hallucination) and the sampling noise.

2. Sequential OSLA Allocation

V0.5 doesn't just treat the number of rollouts as a fixed hyperparameter. It uses One-Step-Look-Ahead (OSLA) to ask: "Will one more rollout reduce my baseline uncertainty more than it costs in compute?"

Figure 1: Comparison between PPO, GRPO, and V0.5. V0.5 uses a frozen V0 model to provide an instant prior, then adaptively adjusts compute based on real-time hypothesis testing.

Experiments: More Stable, More Creative

The researchers tested V0.5 across six brutal math benchmarks. The most compelling evidence for the method's superiority is found in the Gradient Norm and Entropy charts.

• Gradient Stability: In Figure 3 (below), V0.5 maintains a significantly lower and smoother gradient norm compared to GRPO. By reducing baseline MSE, they prevent the "gradient spikes" that often derail training in large-scale RL.
• Sustained Exploration: Because the gradients are less noisy, the model doesn't "panic" and collapse into a single solution. Figure 4 shows that V0.5 keeps policy entropy higher for longer, allowing the model to explore more diverse reasoning paths.

Figure 2: V0.5 (green) shows much tighter gradient stability and higher entropy retention than vanilla GRPO (blue).

Results under Extreme Sparsity

Even with only 4 rollouts, V0.5 achieves performance that GRPO cannot match even with 16 rollouts. This represents a 4x improvement in training efficiency while simultaneously reaching higher absolute accuracy (see Figure 5 in the paper).

Figure 3: Accuracy vs. Group Size. V0.5 scales elegantly where other methods fail to converge when data is sparse.

Critical Insight: The "Group Size 4" Limit

An intriguing theoretical finding in the paper is the derivation of $k\ge 4$ as a fundamental limit. The authors show that in a binary reward space (Correct/Incorrect), any group size smaller than 4 makes it statistically impossible for a hypothesis test to distinguish between "random luck" and "model hallucination." This provides a new "physics limit" for future researchers attempting to optimize RL compute.

Conclusion and Future Work

V0.5 proves that we don't need to choose between the memory-heavy PPO and the variance-heavy GRPO. By treating baseline estimation as a dynamic filtering problem using a pre-trained generalist prior, we can train smarter, not harder.

Future Outlook: The authors plan to extend this to Process-level Generalist Value Models. Imagine a value model that can provide feedback on every single step of a 100-step reasoning chain without the actor ever needing to train its own critic. That is the trajectory V0.5 has set.