Beyond Distribution Sharpening: Why Task Rewards are the Real Engine of RL

TL;DR

Is RL just a fancy way to make models "more sure" of what they already know, or does it actually teach them something new? This paper settles the debate: Distribution Sharpening (increasing confidence) is a dead-end for training stability. Through a unified framework using Llama-3 and Qwen models, the authors prove that Task-Reward signals are the primary driver of performance and stability, particularly when the problems get tough.

The "Sharpening" Identity Crisis

In the world of LLM post-training, there are two schools of thought:

1. The Skill Acquisition View: RL helps models acquire complex reasoning and planning capabilities through feedback.
2. The Distribution Sharpening View: RL just concentrates probability mass on the model's existing high-likelihood modes (essentially a trainable version of Beam Search).

While inference-time tricks like Power Sampling suggest that sharpening works at a surface level, this paper uncovers a darker truth: when you try to train a model to sharpen its distribution, it frequently collapses.

Methodology: A Unified Arena

To isolate the variables, the authors used a KL-regularized RL objective: $\arg {\max }_{heta}\mathbb{E}[r(\hat{y},x)-\beta D_{KL}({\pi }_{h}eta||{\pi }_{ref})]$

By swapping the reward $r$ and the target ${\pi }_{ref}$, they simulated four distinct training philosophies in the exact same codebase:

• Task-Reward RL: Focuses purely on getting the math right.
• Distribution Sharpening: Encourages the model to mimic its own highest-likelihood outputs.
• Tilted & Tempered Sampling: Hybrid approaches adjusting the "temperature" of the distribution during training.

The Core Discovery: The Instability of Sharpening

The most striking finding is expressed in Observation 2: the optima for distribution sharpening is fundamentally unfavorable.

Why? Because LLMs generate variable-length text. Since every token added reduces the total log-likelihood (as probabilities are $\le$ 1), a model tasked with "maximizing likelihood" (sharpening) quickly learns a "cheat code": be as short as possible. This leads to catastrophic collapse where the model stops generating meaningful answers to maximize its objective.

Figure: Note how sharpening-based RL (Dist Sharpen) initially improves but then crashes (Last vs Early Stopped), whereas Task-Reward RL remains stable.

Why Task Rewards Save the Day

When the model is rewarded for the correctness of its mathematical answer rather than its internal token likelihood, the training becomes remarkably robust. On hard datasets like AIME 2024, distribution sharpening methods (both training and inference-time) failed to match the gains of Task-Reward RL.

Key Insight: Sharpening can only elicit what is already "near the surface." For hard tasks where the base model is uncertain, the external reward signal provides the necessary "tilt" to guide the model toward valid solutions it wouldn't have discovered through self-reflection alone.

Figure: Task-Reward RL consistently scales better across different sample counts (k) compared to sharpening baselines.

Critical Analysis & Conclusion

This work provides a necessary theoretical correction to the industry's recent fascination with "self-improvement." The authors explicitly demonstrate that:

1. Optimization matters: RL exposes the flaws in objectives that inference-time hacks (like Beam Search) hide.
2. Environment feedback is king: For frontier models to act as agents, we cannot rely on entropy reduction alone; we need verifiable, task-specific rewards.

Limitation: The study focuses on verifiable math tasks. Whether these conclusions hold for non-verifiable tasks (like creative writing) where rewards are derived from potentially noisy Reward Models (RMs) remains an open question for future work.

Final Takeaway: If you want a model that reasons better, stop trying to make it more "confident" in its own shadow. Give it a ground-truth reward and let RL do the heavy lifting of capability discovery.