[Preprint 2026] MOGA: Rethinking Hyperparameter Transfer via Mean-Normalized Matrix Operator Norms

TL;DR

Scaling laws tell us that bigger models are better, but they don't tell us how to tune them. MOGA (Matrix-Operator-Geometry-Aware) optimization bridges this gap by proving that if we use "mean-normalized" operator norms, we can achieve width-independent smoothness ($L$) and Lipschitz ($M$) constants. This allows a learning rate tuned on a tiny model to work perfectly on a massive one, outperforming AdamW and Muon in large-scale LLaMA and GPT-2 pre-training.

The Problem: The "Hyperparameter Cliff" in Scaling

In modern AI, tuning a 70B parameter model is prohibitively expensive. We usually tune a "proxy" model (e.g., 100M) and hope the learning rate transfers. However, in standard Euclidean geometry (Frobenius norm), the curvature of the loss landscape—its smoothness—scales with the hidden dimension $w$.

The authors identify a critical flaw: classical operator norms ($poq$) fail to propagate stability because they aren't "compatible" across layers. As activations pass through a network, the geometric mismatch amplifies perturbations, making the optimal learning rate highly dependent on the model's width.

The Mathematical Insight: Mean-Normalization

The core contribution is the shift from standard ${\ell }_p$ norms to mean-normalized norms: $\Vert \mathbf{x}{\Vert }_{(p,extmean)}:={\left(\frac{1}{n}{\sum }_{i=1}^n|x_i{|}^p\right)}^{1/p}$

By introducing the $1/p$ scaling, the authors ensure that the "size" of a vector doesn't explode just because the dimension increases. This leads to the Compatibility Condition: $\Vert \cdot {\Vert }_{extin}\le \Vert \cdot {\Vert }_{extout}$

When this condition holds between consecutive layers, the Lipschitz constant of the whole network becomes independent of width. The paper proves that while Muon ($2o2$ norm) has a smoothness constant that grows with $O(\sqrt{w})$, Row Normalization under the mean-normalized geometry remains $O(1)$—perfectly stable.

Methodology: The MOGA Optimizer

MOGA translates these geometric insights into practical update rules. Depending on the chosen geometry, the update is scaled by dimension-dependent factors that "absorb" the width dependence into the step size.

Figure 1: Geometric compatibility between layers. The (p, mean) geometry ensures dimension-independent stability bounds.

For a given gradient $G$, the MOGA Row Normalization update is: $W_t=W_{t-1}-\eta \cdot \frac{1}{d_{extin}^{1/p}}\cdot ext{rownorm}_p(G)$

Experimental Proof: Zero-Shot Transfer

The most striking result is the learning rate transfer. In GPT-2 models ranging from Small (124M) to XL (1.5B), the "U-curve" of the learning rate vs. loss stays perfectly aligned.

Figure 2: MOGA (p=3) demonstrating that the optimal learning rate is width-invariant.

Key highlights from the results:

• Stability: Unlike Muon, MOGA doesn't suffer from rough curvature in later stages of training.
• Efficiency: On LLaMA-130M, MOGA matched Muon's speed but showed a steeper downward trend in the final low-loss regime.
• Generality: The theory explains why $\mu$P work for Adam but also provides new rules for row/column normalization that $\mu$P couldn't originally justify.

Critical Analysis & Conclusion

MOGA provides a rigorous "Matrix Thinking" framework for optimization. It suggests that the industry's reliance on AdamW might be suboptimal not just because of preconditioning, but because of geometry.

Limitations:

• The analysis assumes parameters stay within a bounded set (though weight decay usually handles this).
• The "optimal" $p$ and $q$ settings introduce a trade-off between optimization speed and the network's approximation capacity (expressivity).

Final Takeaway: To scale models reliably, we must look at the operator norms of the layers. MOGA's row-normalization approach offers a principled way to ensure that as our models grow wider, our hyperparameters don't need to change.