Rethinking the Divergence Regularization in LLM RL

DRPO: Smoothing the Path for LLM Reinforcement Learning via Divergence Regularization

Rethinking the Divergence Regularization in LLM RL

2026-06-01

Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, Tianyu Pang

Summary

Problem

Method

Results

Takeaways

Abstract

The paper introduces Divergence Regularized Policy Optimization (DRPO), a novel reinforcement learning (RL) objective for Large Language Models (LLMs). It replaces the standard ratio-based clipping in PPO/GRPO with a smooth quadratic regularizer based on Binary Total Variation (Binary-TV) to ensure stable off-policy optimization across various architectures and precisions.

TL;DR

Reinforcement Learning (RL) for LLMs is notoriously unstable due to "off-policy" discrepancies between training and inference engines. Divergence Regularized Policy Optimization (DRPO) solves this by ditching the brittle "importance sampling ratio" used in PPO for a smooth regularizer based on absolute probability shifts. This shift provides a bounded, corrective gradient that prevents training collapse—even in extreme low-precision (FP8) environments—and sets new standards for stability in mathematical reasoning tasks.

Problem & Motivation: The "Long-Tail" Ratio Trap

Most mainstream RL algorithms like PPO and GRPO use a ratio-based trust region. The logic is simple: keep the ratio of current policy to behavior policy ( $π / μ$ ) near 1.0.

However, the authors point out a fundamental flaw: LLM vocabularies are vast and long-tailed.

For a rare token (prob: $1 0^{- 5}$ ), a tiny absolute change can cause a massive ratio spike ( $10 im es$ ), triggering unfair gradient clipping.
For a frequent token (prob: $0.9$ ), a massive shift to $0.7$ results in a modest ratio change ( $0.77 im es$ ), potentially escaping the trust region unnoticed.

Prior work like DPPO attempted to fix this with a "hard mask" based on probability shift, but it suffered from "gradient death"—once a token crossed the line, it provided no feedback on how to get back.

Methodology: The Core Logic of DRPO

DRPO effectively marries the smooth stability of Simple Policy Optimization (SPO) with the robust geometry of DPPO.

Instead of regularizing the ratio, DRPO applies an advantage-weighted quadratic penalty to the absolute probability shift ( $∣ π - μ ∣$ ). The gradient of the DRPO objective introduces a continuous weight $w_{t}$ :

$w_{t} = 1 - e x t s i g n (\hat{A}_{t} (r_{t} - 1)) \frac{D _{t}^{e x t B in - T V}}{δ}$

Why this works:

Correction, not just Clipping: When the policy drifts too far (outside $δ$ ), the gradient weight becomes negative, literally pulling the model back toward the behavior policy.
Bounded Weights: Unlike SPI/SPO, where weights can explode toward infinity in the "long tail" (low $μ$ ), DRPO's weights are strictly bounded, ensuring no single token can hijack the entire gradient update.

Model Architecture and Comparison Concept Graph: Note how DRPO's weight (bottom right) remains bounded across all probabilities, unlike SPO (top right) which spikes at low probabilities.

Experiments & Results: Stability Meets SOTA

The team tested DRPO across various scales (1.5B to 35B parameters) and precision settings (BF16, FP8).

Key Findings:

FP8 Resilience: In "End-to-End FP8" training—a setting that usually causes RL to collapse—DRPO remained perfectly stable while GRPO and SPO failed.
Superior Convergence: On the AIME (American Invitational Mathematics Examination) benchmarks, DRPO reached higher peak accuracy faster than its predecessors.

Experimental Results Comparison Figure: DRPO consistently maintains a superior accuracy curve across various model backbones compared to hard-clipped (GRPO) or unregularized methods.

Deep Insights: The "Gradient-First" View

A crucial takeaway from this research is the Ablation on Advantage Weighting. Many researchers use a pure KL penalty, but the authors found that weighting the penalty by the absolute advantage ( $∣ \hat{A}_{t} ∣$ ) is vital. This ensures the trust-region boundary stays fixed regardless of whether the reward is +1 or +100.

Ablation on Advantage Weighting Ablation Study: Removing the advantage weight (orange) leads to significant performance degradation compared to the full DRPO (blue).

Conclusion

DRPO represents a shift in thinking from "Which divergence should I minimize?" to "What does the gradient weight look like?" By choosing an $L_{2}$ -style penalty on probability space, DRPO provides a robust, smooth, and corrective framework for LLM RL. It is particularly valuable for groups training at scale with low precision (FP8/FP4), where numerical noise makes traditional ratio-clipping too brittle.

Limitations: While the Binary-TV approximation is efficient, it only looks at the sampled token. Future work might explore "Top-K TV" for even more precise distributional control.

Find Similar Papers

Try Our Examples

Search for recent papers dealing with the training-inference mismatch in off-policy RL for LLMs, specifically focusing on alternatives to importance sampling ratios.
Identify the origin of the Binary Total Variation (Binary-TV) concept in LLM trust regions and how it compares to standard KL-divergence penalties.
Are there studies applying smoothed quadratic divergence regularizers to multimodal models or reinforcement learning from human feedback (RLHF) for non-reasoning tasks?

Contents

DRPO: Smoothing the Path for LLM Reinforcement Learning via Divergence Regularization

1. TL;DR

2. Problem & Motivation: The "Long-Tail" Ratio Trap

3. Methodology: The Core Logic of DRPO

3.1. Why this works:

4. Experiments & Results: Stability Meets SOTA

4.1. Key Findings:

5. Deep Insights: The "Gradient-First" View

6. Conclusion