How does decoupling update magnitude from gradient norm affect optimizer scaling?

Normalized SGD turns shrinking gradients into a rising learning rate

Modern adaptive optimizers such as RMSProp, Adam, SOAP and Muon rescale updates through gradient-dependent preconditioning even when the base learning rate is fixed. To isolate that magnitude control from direction, this study analyzes normalized SGD, which keeps the stochastic-gradient direction but fixes the update norm. With the normalization used here, it is equivalent to SGD with an endogenous scalar learning rate ηeff(t) = η / ‖g(t)‖₂.

That equivalence is the whole mechanism. As progressively weaker spectral modes are learned, the gradient norm decays, so the effective learning rate grows. The paper frames this as a tension: the spectral tail controls transient acceleration, while the spectral edge controls when that acceleration breaks down.

Acceleration maps the SGD exponent to 2r/(1−r)

The paper studies normalized SGD in a random-feature model with power-law teacher and data covariance, and derives a dynamical mean-field theory (DMFT) for how loss depends jointly on training time, model width and batch size. Within that model, normalization initially accelerates SGD by mapping the power-law exponent rSGD < 1 to 2rSGD/(1 − rSGD).

The same mapping gives two boundary cases. At rSGD = 1 the theory predicts exponential convergence, and for rSGD > 1 it predicts formal finite-time convergence. These are transient predictions of the continuous-time flow, not the final behavior of the discrete optimizer.

At finite step size the same feedback breaks the acceleration

The acceleration does not persist. At finite step size, the growing effective learning rate eventually brings the leading spectral mode to the discrete-time stability threshold, and the dynamics depart from the flow prediction. The deterministic edge-of-stability (EoS) corresponds to marginal stability of the leading Hessian mode, while the edge-of-stochastic-stability (EoSS) is a stability limit imposed by minibatch fluctuations.

The paper's random-matrix analysis makes the EoS condition concrete: the effective learning rate satisfies ηeff = 2/λmax, where λmax is the largest eigenvalue of Λ^(1/2) Aᵀ A Λ^(1/2). So the transient acceleration and its breakdown come from different parts of the spectrum — the tail drives the gain, the edge ends it.

Late time splits into width-limited, EoSS and EoS regimes

Solving the DMFT for long times, the authors assume the effective learning rate settles to a steady value η∞eff and find three candidate steady states, corresponding to three phases. Phase 1 is width-limited, Phase 2 is edge-of-stochastic-stability, and Phase 3 is deterministic edge-of-stability.

In overparameterized models, a width-dependent critical batch separates EoSS from EoS; underparameterized models are instead limited by finite-width error. The paper also extends the analysis to a dimension-free limit where the dimension drops out of every equation and only N, B, η and the spectrum remain. In that limit Phase 1 and Phase 3 survive but Phase 2 does not: the width-limited state is reached first for every N, B and η, leaving only a width-limited-to-EoS transition at a critical batch size.

Only some phases convert extra resources into fewer steps

The phases determine when larger batches or wider models reduce serial training time at comparable compute. The paper quantifies in which phase increased batch size or width can compensate the excess compute use per step by requiring fewer optimization steps to reach a target loss.

The negative results are as specific as the positive ones. In Phase 1, at fixed width and spectrum, the width constraint fixes η∞eff ρ, so the stationary effective learning rate is independent of batch size; the small-step asymptotics give ∂log η∞eff / ∂log α → 1/2, so doubling the batch reduces the updates needed for the same effective time only by a factor √2 and increases total compute by √2. In Phase 2, at fixed batch size and spectrum, the EoSS condition fixes y independently of width, and the response equation gives ∂log η∞eff / ∂log ν = α/(ν+α) < 1/2, so width gains are slower than square-root. In Phase 3, the deterministic threshold η∞eff = 2/λmax(ν) is independent of batch size, so additional batching provides no stationary speedup.

The paper's positive claim is narrower. In the EoSS phase, a nearly linear batch-scaling regime reduces the steps to a target loss at almost unchanged compute. In the width-limited phase, the authors identify spectral regimes where the stationary effective learning rate grows superlinearly with width, giving potential width speedups — but this gain is eventually reversed by stability on the EoS branch, and when that branch is reached below ν = 1 the increasing and decreasing regimes imply an efficiency maximum at an underparameterized width.

Linearized ResNets on CIFAR-5M support the predicted trends

The theory is tested with linearized ResNet experiments on CIFAR-5M, using spectral exponents measured from the neural tangent kernel and target alignment. The measured exponents yield rSGD = 0.095, with predicted transient exponent 2rSGD/(1 − rSGD) = 0.21.

The DMFT takes the measured spectrum for k ≤ 1024 and a fitted power law beyond it, continued to d = 10^10; taking larger d made a negligible difference. The sweep varies width over 4, 16, 64, 128 and 256 (N from 1,120 to 4,135,936), batch size over 2, 10, 100, 1,000 and 10,000 (T from 2.5·10^6 down to 500), and learning rate over 0.01, 0.1, 0.2 and 0.5, with two seeds per cell for 241 runs in total. Evaluation reports mean-squared error on a fixed held-out set of 4,096 images at 150 logarithmically spaced steps, recording ‖gt‖₂ and ηteff at every step; the DMFT is integrated up to T = 3800 steps due to memory constraints.

The experiments support the predicted acceleration, breakdown and resource-scaling trends in the explored regime. They are linearized-network experiments, so they test the theory's spectral predictions rather than full end-to-end training of large networks.

Judge batch and width gains by phase, not by resource size

The practical content of this paper is a phase split, not a general endorsement of bigger batches or wider models. The same resource increase can be nearly compute-neutral, slower than square-root, or stationary-useless depending on whether the system sits in the EoSS phase, the width-limited phase, or the deterministic EoS phase.

The authors are explicit that norm-SGD is used to isolate the feedback between gradient normalization and learning dynamics, not advocated as a practical optimizer. So the transferable claim is about the mechanism: when update magnitude is decoupled from gradient norm, the effective learning rate is an endogenous quantity that grows as gradients shrink, and the compute value of batch size or width depends on which stability limit that growth reaches first. The most decision-relevant threshold in the paper is the width-dependent critical batch that separates EoSS from EoS in overparameterized models, together with the finding that in the dimension-free limit only the width-limited-to-EoS transition remains.

Sources used in this answer