How does generalized preference optimization unify offline alignment algorithms?

One convex loss family covers DPO, IPO and SLiC

The paper's central move is to stop treating offline alignment algorithms as separate recipes. It defines a single loss that averages a convex function f over preference pairs, where the function is applied to a log-ratio difference: the log-probability ratio of the preferred response under the trained policy and the reference model, minus the same ratio for the dispreferred response. The trained policy is the model being aligned; the reference model is the fixed starting point that the loss is measured against.

Within this family, the named algorithms differ only in which convex function is used. DPO applies a scaled logistic loss, SLiC applies a hinge loss, and IPO applies a squared loss. The paper presents these as convex approximations to the 0-1 loss that measures binary classification accuracy, which is why the derivation can borrow from supervised binary classification rather than from reinforcement learning. Any other convex alternative supplies an offline preference loss that is not yet in the existing literature, which is how the framework generates new variants rather than only relabeling old ones.

The classification view explains what the policy learns

Framing reward modeling as binary classification does more than organize the algebra. Under the paper's Bayes-consistency discussion, a convex, continuously differentiable f with f′(0) < 0 yields a minimizer whose sign matches the Bayes-optimal classifier, so the learned preference prediction gets the direction of the preference right. That result is stated for a general bivariate preference model, and the paper explicitly leaves the transfer to pointwise reward models, where the Bradley-Terry assumption only approximately holds, to future work.

Getting the sign right is not the same as getting the magnitude right. The paper notes that even when different losses produce the same sign, the predicted values can differ drastically depending on f, and it offers a logistic-versus-hinge case study as an illustration. This distinction matters for readers who might assume that any Bayes-consistent convex loss must produce interchangeable trained policies.

Offline regularization only approximates the KL penalty

Canonical RLHF maximizes reward minus a KL divergence term between the trained policy and the reference policy. GPO losses do not contain that term directly; instead, the choice of convex function determines how strongly the offline loss pulls the policy back toward the reference. The paper's analysis and experiments show that offline regularization and the intended KL regularization are different in general, with connections and subtle differences rather than an identity.

The authors do derive a concrete link. The gradient of the KL divergence equals the gradient of a squared loss weighted by the preference-pair distribution, when that distribution is set to the policy itself. This is why the squared loss, which is effectively the IPO loss, plays a special role in the regularization story: it is the variant whose gradient matches KL under that weighting. The paper also shows that a second-order Taylor expansion of any smooth convex GPO loss with f′(0) < 0 recovers the squared loss with a rescaled beta, so the squared loss is a local approximation to the whole family.

Beta sets regularization strength differently per variant

The trade-off experiments use a summarization task with human feedback labels, a T5X encoder-decoder base model with 700M parameters, a constant learning rate of 1e-5, batch size 32, the Adafactor optimizer with decay 0.8, and 2×10^5 training steps. Evaluation is a side-by-side comparison judged by a prompted PALM-2 model on 2000 examples, with the supervised fine-tuned model as the default baseline. These conditions define the scope: one task, one model family, one automated judge.

Across variants, peak win rate against the supervised fine-tuned baseline is similar, but the beta at which the peak occurs is not. Squared and truncated squared losses peak at generally lower beta around 1, while the other variants peak at higher beta around 10. The paper attributes this to distinct regularization strengths induced by each convex function as a function of beta, and reports that at large beta the correlation between offline regularization and KL divergence becomes much stronger, with squared and truncated squared enforcing stronger regularization than logistic, exponential and Savage, and hinge in the middle.

At a shared beta, the variants perform about the same

The head-to-head comparison fixes beta at 0.1, a value the paper says lets all algorithms work reasonably according to win rate against the supervised fine-tuned checkpoint. In that matrix, the win rates of one GPO variant against another cluster near 50%, for example 50%, 48%, 50%, 47%, 50% and 48% in the first row. The paper reads this as the variants performing mostly similar, with some variation in peak win rate, such as exponential appearing slightly better, that may not be statistically significant.

The practical reading is a split between choosing a loss and tuning its strength. If variants are compared at a matched beta, the differences are small in this setting; if each variant is tuned to its own best beta, the peak performance is also similar but the beta that achieves it differs, as theory predicts. That is the paper's most decision-relevant comparison for a practitioner deciding whether a new convex loss is worth adopting.

The framework covers contrastive offline losses only

The paper states two boundaries of its own. The framework builds on the reward-maximization formulation of RLHF, so it still inherits theoretical issues when the ground-truth preference structure is complex; the authors suggest connecting GPO to alternative solution concepts such as Nash equilibrium as future work. It also handles only offline losses with a contrastive form and does not cover supervised-learning-based losses.

These limits sit next to the results they qualify. The unification claim is about contrastive offline preference losses parameterized by convex functions, not about every alignment objective, and the empirical trade-off findings come from the single summarization setup described above rather than from a broad benchmark sweep.

Pick beta by the loss, not by habit

The paper's most useful priority for alignment practitioners is that beta is not a portable hyperparameter across GPO variants. Because each convex function induces a different regularization strength, a beta that works for a logistic-style loss is not the beta that works for a squared loss; the experiments show squared and truncated squared peaking near beta around 1 while other variants peak near beta around 10. Tuning beta per loss, rather than transferring a value from DPO or IPO practice, is the concrete step the evidence supports.

The second priority is expectation-setting. At a matched beta the variants are close, and at their own best beta their peak win rates are also close in this controlled summarization setting, so a new convex loss is unlikely to be a large performance jump on its own. Its value lies in the regularization behavior it induces and in the option to choose among convex functions deliberately, which is what the unified view makes visible.

Sources used in this answer