Can single-pass backpropagation learn full Pareto stationary fronts efficiently?

Pareto stationarity is a gradient condition, not a single best model

Multi-objective optimization in deep networks starts from m non-negative, differentiable losses over the same parameters. Because those losses usually conflict, no single parameter setting minimizes all of them, so the paper works with local notions of optimality: a point is locally Pareto optimal if no nearby point improves one objective without worsening another, and the image of that set is the local Pareto front.

The paper's practical target is the weaker, checkable condition of Pareto stationarity. A point is Pareto stationary when some convex combination of the loss gradients vanishes, that is, there exist non-negative weights summing to one whose weighted gradient sum is zero. Pareto stationarity is necessary for local Pareto optimality and becomes sufficient under convexity assumptions, which is why the paper reports fronts of Pareto stationary solutions rather than claiming global optima.

This distinction matters for reading every result below. Hypervolume and the count of Pareto optimal solutions describe how much of the trade-off surface a method recovers, not whether it found a unique optimum. A method can look strong on one of these measures and weak on the other, and the paper reports both.

Scalarization recovers one trade-off per training run

A common way to handle conflicting losses is scalarization, which collapses the vector problem into a single surrogate objective encoding a preference vector. The paper examines two variants: linear scalarization, a weighted sum of losses, and smooth Tchebychev scalarization, which uses a smoothing parameter and the utopia point, the ideal per-objective value attainable in isolation and generally infeasible when objectives conflict.

The two are not interchangeable. Linear scalarization can only recover solutions on convex parts of the Pareto front, while Tchebychev scalarization does not suffer from that limitation. This is why the paper carries both MOSEL-LS and MOSEL-CS through its experiments instead of treating scalarization as a single design choice.

The cost problem is structural. A priori methods compute one Pareto stationary solution for a fixed preference and require retraining for each new preference, whereas a posteriori methods aim to generate a set of trade-off solutions. MOSEL is positioned as an a posteriori method whose selling point is producing that whole set without paying the retraining cost per preference.

MOSEL splits representation learning from preference alignment

MOSEL reformulates a posteriori multi-objective optimization as a bilevel problem that exploits network modularity to decouple representation learning from objective-preference alignment. Casting that bilevel problem as a Stackelberg game is what allows the original a posteriori problem to be solved in a single forward-backward pass.

The efficiency claim rests on a timescale separation approximation. It reduces the nested second-order derivatives of exact bilevel optimization to standard gradient descent, so Pareto stationarity is reached without extra computational cost. The paper states that MOSEL shares the same asymptotic complexity as COSMOS, with O(e·m·p) time and O(e·p) memory, where e is the number of epochs, m the number of objectives, and p the number of network parameters, because both methods perform a single forward and backward pass over m objectives per gradient step.

The split architecture is not free in parameters, but the overhead is small and measured. On the multi-task benchmarks, MOSEL-LS and MOSEL-CS add 4,320 parameters relative to equal-weight training, against 708 for COSMOS, while several baselines add none. Relative time is reported as roughly 1.1 to 1.4 times equal-weight training across the three multi-task datasets, with GPU memory ratios near 1.0.

On fairness-accuracy, MOSEL matches or beats COSMOS

The strongly conflicting setting is fairness versus predictive performance, where metrics such as Difference in Equality of Opportunity directly conflict with accuracy. The paper evaluates on Adult, COMPAS, and Credit, three tabular fairness benchmarks using sex as the sensitive attribute, plus the image dataset FairFace predicting gender under race as the sensitive attribute. Training objectives are binary cross-entropy as an accuracy proxy and a differentiable hyperbolic tangent relaxation of DEO as a fairness surrogate; test time reports misclassification error and DEO.

Against COSMOS, evaluated on 25 uniformly distributed test rays, MOSEL-LS and MOSEL-CS match or outperform it on both hypervolume and the number of Pareto optimal solutions across datasets. The gap is most visible on FairFace, where COSMOS collapses to a near-degenerate front with relative hypervolume 0.193 on training loss and 0.176 on test error, while MOSEL-LS reaches 0.987 and 0.701 and MOSEL-CS reaches 0.998 and 0.734.

The post-processing baseline of Hardt et al. remains competitive in hypervolume in some cases, such as Default and FairFace test error, but generally yields fewer non-dominated solutions. On Adult, its solutions concentrate in a narrow region of low unfairness but high error, which the paper frames as limiting practical utility. The relevant comparison is therefore not only peak hypervolume but how much of the trade-off surface a decision maker can actually inspect.

In multi-task learning, MOSEL moves closer to the utopia point

Multi-task learning sits in a weakly conflicting regime: joint training can improve tasks through positive transfer, but task-specific losses may also induce conflicting gradients. Because conflicts are not guaranteed, methods designed to resolve gradient conflicts, including GradNorm, FAMO, NashMTL, and PCGrad, can even harm performance. The paper uses this regime to argue that MOSEL adapts to both regimes without prior knowledge of the conflict structure.

In the two-task setting, MOSEL converges to solutions closer to the utopia point by leveraging positive transfer, outperforming both standard single-objective training and specialized multi-task learning methods without additional computational cost. The test-time Pareto fronts plot task-specific errors, with grey dotted lines marking the best achievable single-task performance on each task and their intersection marking the single-task baseline; positive transfer lets methods surpass that baseline, and the paper reports MOSEL achieving the largest improvements.

Scalability is tested on CelebA, with over 200,000 celebrity images annotated with 40 binary attributes. Per-task accuracy improvement over the single-task baseline is reported for the most correlated and least correlated task sets as the number of jointly optimized tasks grows from 1 to 10, averaged over 5 seeds. The paper also reports that runtime stays roughly constant and peak memory only slightly increases as the number of objectives rises from 2 to 10, with memory identical between correlated and uncorrelated feature settings.

The Dirichlet parameter trades hypervolume against front spread

MOSEL samples preferences stochastically, and the Dirichlet parameter α controls how concentrated those samples are. The ablation shows the effect is regime-dependent. In strongly conflicting settings, performance degrades substantially as α increases, particularly under linear scalarization: on Adult, hypervolume drops from 0.939 at α=0.5 to 0.816 at α=1.2, while on COMPAS the decline runs from 0.920 to 0.164, indicating near-total failure to discover diverse trade-off solutions when preference samples concentrate near the simplex center.

Under Tchebychev scalarization the picture differs. Adult follows a similar downward trend, from 0.819 to 0.524, whereas COMPAS stays remarkably stable with hypervolume above 0.920 across all α values. The two multi-task datasets are largely insensitive to α: Multi-Fashion is effectively constant, and Multi-Fashion+MNIST shows only a mild upward trend as α increases.

The paper draws two conclusions. First, α matters most where objectives strongly conflict and where the scalarization function is sensitive to the geometry of the preference simplex, with linear scalarization more sensitive than Tchebychev. Second, even where hypervolume stays stable, increasing α systematically contracts the extreme solutions and reduces front diversity, an effect aggregate metrics can obscure. The paper recommends α=0.5 as a robust default, achieving best or near-best hypervolume across all dataset-scalarization combinations and consistently recovering the widest spread of extreme solutions.

The single-pass claim holds across MLPs, CNNs, and DistilBERT

The efficiency claim is tested beyond the main benchmarks. On fairness-aware text classification, MOSEL is evaluated with DistilBERT on a 100k-sample subset of CivilComments, optimizing toxicity prediction and DEO under linear scalarization with α=0.5. DistilBERT has six Transformer blocks, and three candidate split points are evaluated. MOSEL extends to this architecture without architectural modifications, achieving consistent Pareto-front quality across the three splits with hypervolume 0.419 to 0.423 and standard deviation at most 0.011.

The paper reads this as evidence that MOSEL does not rely on architecture-specific decompositions but instead leverages the modular structure common to modern neural networks by partitioning them at natural block boundaries such as residual stages or Transformer blocks. Extending the empirical validation to foundation vision-language models and large language models is left as future work.

Two boundaries qualify the results. The method relies on a modular decomposition of neural networks, which may not be straightforward to design or optimal for all architectures, and exploring adaptive or learned decompositions remains open. Separately, deploying the model requires selecting a preference vector and therefore a specific trade-off, a decision the paper says becomes increasingly challenging as the number of objectives grows and remains an open problem.

Compare front diversity, not just peak hypervolume

The paper's own results point to a specific reading priority. On FairFace, COSMOS and MOSEL differ less in whether they find good trade-offs than in how much of the front they retain: COSMOS collapses to a near-degenerate front while MOSEL keeps a substantially larger set of Pareto optimal solutions. On Adult, the post-processing fairness baseline can be competitive in hypervolume yet concentrates its solutions in a narrow low-unfairness, high-error region. Peak hypervolume alone would not separate these cases.

The second comparison is regime. In strongly conflicting fairness-accuracy settings, the choice of scalarization and the Dirichlet parameter α materially change outcomes, with linear scalarization degrading sharply on COMPAS as α rises while Tchebychev stays stable above 0.920. In weakly conflicting multi-task settings, the same α is largely inert and the relevant question becomes whether positive transfer moves solutions closer to the utopia point than single-task training and specialized multi-task baselines.

The practical threshold the paper supplies is α=0.5, the value it reports as best or near-best across all dataset-scalarization combinations and as recovering the widest spread of extreme solutions. The measured price of the single-pass front is a 4,320-parameter increase and roughly 1.1 to 1.4 times equal-weight training time on the multi-task benchmarks. The open decision that remains is which preference vector to deploy once the front exists.

Sources used in this answer