[ArXiv 2026] MoUE: Flipping Depth into Width – Scaling LLMs via Virtual Capacity
Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
The paper introduces Mixture of Universal Experts (MoUE), a generalized MoE framework that scales model capacity through "Virtual Width" by reusing a shared pool of layer-agnostic experts across different depths. MoUE achieves SOTA performance on multiple benchmarks, outperforming standard MoE baselines by up to 1.3% in width expansion and 4.2% in progressive conversion of existing checkpoints.
TL;DR
Standard Mixture-of-Experts (MoE) models are physically "rigid"—to get more capacity, you need more memory. Mixture of Universal Experts (MoUE) breaks this bottleneck by introducing Virtual Width. By reusing a shared pool of "Universal Experts" across different layers, MoUE turns network depth into combinatorial capacity. Even with a fixed parameter budget, MoUE significantly boosts accuracy (up to +4.2% on existing checkpoints) by enabling tokens to find complex, reusable pathways through the model.
The "Physicality" Problem in MoE Scaling
Scaling Laws dictate that bigger is better, but in the world of MoE, "bigger" usually means a linear increase in VRAM for expert storage. Furthermore, standard MoEs treat each layer as a discrete island; an expert learned in Layer 2 cannot be accessed by Layer 20, even if the functional requirement (e.g., "Logic Reasoning" or "Python Syntax") is identical.
The authors' insight is grounded in Functional Redundancy: their CKA analysis (Centered Kernel Alignment) reveals that experts across different layers often learn highly similar functions. If we can reuse these functions, we can create "Virtual Width"—an exponential increase in the number of possible routing paths without adding a single physical parameter.
Methodology: The Architecture of Virtual Width
Reusing experts across layers is theoretically powerful but practically difficult due to "routing path explosion." MoUE solves this with a three-pronged system.
1. Staggered Rotational Topology
Instead of an unconstrained "all-to-all" expert access, MoUE organizes experts on a logical ring. As the data flows deeper into the model, the "window" of accessible universal experts shifts (rotates). This ensures local specialization while maintaining global connectivity.
Figure 1: Comparison between standard MoE and MoUE's shared Universal Expert pool.
2. Universal Expert Load Balance (UELB)
Standard load balancing (Switch Transformer style) fails in MoUE because "Universal Experts" are seen by more layers than "Local Experts." This "Heterogeneous Exposure" causes the model to over-penalize shared experts. UELB introduces a normalization factor based on the expert's topological degree (how many layers can see it), ensuring fair optimization.
3. The Universal Router
To prevent the model from getting lost in the "Virtual Width," the router needs memory. The Universal Router maintains a lightweight "trajectory state" (fast-weights) that allows routing decisions at Layer to be coherent with decisions made at Layer .
Experimental Performance: Shifting the Frontier
The results confirm that MoUE is a "free lunch" in terms of capacity.
- Width Expansion: By simply converting existing experts into a universal pool (Virtual Width), accuracy increased from 43.8 to 44.7 on a 700M scale model with zero additional parameters.
- Depth Expansion: MoUE allows models to become deeper (more computation steps) while sharing the FFN parameters. A MoUE with 36 layers (L36) outperformed a standard 64-expert MoE while using only 50% of the total physical parameters.
Figure 2: Scaling trends showing MoUE (stars/triangles) consistently residing on a superior frontier compared to standard MoE.
Why It Works: Spontaneous Specialization
Analysis of the expert utilization heatmaps shows that Universal Experts don't just become "generic" filters. They develop Domain Specialization. Some experts become the model's dedicated "Math" units, while others specialize in "Code," and they are called upon repeatedly at different stages of the Transformer's depth to refine the hidden state.
Figure 3: Visualization of experts specializing in different domains across the universal pool.
Conclusion & Insights
MoUE represents a fundamental shift in how we think about model dimensions. By decoupling physical storage from functional connectivity, it suggests that our current LLMs are likely parameter-redundant and depth-inefficient.
The "Progressive Warm-Start" capability is particularly valuable for the industry; it allows existing SOTA MoE checkpoints (like OLMoE) to be "upgraded" to MoUE architectures via continued training, unlocking significant performance gains without starting from scratch.