UniPool: Breaking the Per-Layer Barrier in Mixture-of-Experts

TL;DR

Traditional Mixture-of-Experts (MoE) models are architecturally rigid: each layer owns its experts, forcing parameter counts to explode linearly with depth. UniPool shatters this convention by introducing a globally shared expert pool. By allowing all layers to access the same expert bank through a specialized "pool-level" balancing act, UniPool achieves better accuracy than standard MoE while using up to 60% fewer expert parameters.

The "Depth Redundancy" Problem

The motivation for UniPool stems from a startling observation in production MoE models (like Qwen and DeepSeek). In standard designs, each layer has its own "private" experts. However, research shows that in deeper layers, these experts are highly redundant.

The authors conducted a routing-randomization probe: if you replace the learned router in a deep layer with a random one, accuracy only drops by a measly 1.0–1.6 points. This proves that standard MoE models aren't really "specializing" at depth; they are just duplicating effort. Why give every layer its own isolated budget if they all end up doing the same thing?

Methodology: The Global Shared Pool

UniPool flips the script by treating expert capacity as a global resource. Instead of Layer 1 owning Experts {A, B} and Layer 2 owning {C, D}, all layers route their tokens into a single pool {A, B, C, D, ...}.

To make this work, the authors solved two critical technical hurdles:

1. Pool-Level Balancing: In standard MoE, an "auxiliary loss" forces each layer to use all its experts. But in UniPool, one layer might only need a specific subset of the pool. Forcing every layer to use every expert would ruin specialization. UniPool's new loss aggregates utilization across all layers, ensuring the pool is used efficiently without micro-managing individual layers.
2. NormRouter: Different layers have different hidden-state magnitudes. A standard Softmax router might get "confused" by these scale differences when looking at a large global pool. UniPool uses NormRouter, which applies L2-normalization to make routing decisions based on the direction rather than the magnitude of vectors, ensuring stable competition for experts.

Experimental Performance: Sublinear Scaling

The most impressive result is the "Reduced-Pool" efficiency. UniPool doesn't just perform better; it allows the model to be smaller.

As shown in the charts, UniPool variants using only 50% or even 41.6% of the vanilla expert-parameter budget were able to match or outperform the standard layer-wise MoE. This proves that under a shared-pool design, expert parameters do not need to grow linearly with the number of layers.

Critical Insight: Why Does Sharing Help?

In a private MoE, an expert only learns from the gradients of its specific layer. In UniPool, an expert can receive gradients from every layer in the network. This significantly increases the "data-to-parameter" density for each expert, forcing them to become more robust and specialized.

Interestingly, when the authors performed the same random-routing probe on UniPool, the accuracy drop was 4.1 points—much larger than the 1.5 drop in vanilla MoE. This confirms that UniPool's experts are doing more "load-bearing" work; the routers are making choices that actually matter.

Conclusion & Future Outlook

UniPool represents a fundamental shift in how we think about scaling LLMs. By decoupling "Expert Capacity" from "Model Depth," we can build models that are deeper and more expressive without the bloated memory footprint of traditional MoE.

Limitations: While the logic is sound, scaling this to 100B+ parameter models remains to be seen. Additionally, the communication overhead of routing into a massive global pool in a distributed setting (Expert Parallelism) will require careful engineering to maintain high throughput.