SlimQwen: Masterful MoE Compression via Progressive Pruning and MTP Distillation

TL;DR

The Qwen team has unveiled SlimQwen, a specialized recipe for compressing massive Mixture-of-Experts (MoE) models. By transforming the Qwen3-Next-80A3B into a 23A2B variant, the researchers demonstrate that structured pruning across depth, width, and experts—combined with a novel Multi-Token Prediction (MTP) distillation—can retain over 86% of teacher performance while slashing parameter counts by nearly 4x.

Strategic Position: This work moves beyond simple "one-shot" pruning, establishing a gold standard for MoE model compression at the pre-training scale through "progressive schedules."

1. The Search for a Stronger Initialization

A central question in model compression is: Is it better to prune a big model or train a small one from scratch? SlimQwen provides a definitive answer.

Pruning provides a "warm" initialization that bypasses the chaotic early phases of training. Experimental results show a massive +11.79 point lead for the pruned initialization over a target-matched model trained from scratch. Even with a high compression ratio, the inherited weights act as a powerful inductive bias for reasoning and knowledge-intensive tasks.

Figure: The "Pruned + KD" approach (purple) shows the fastest convergence and lowest final loss, proving the efficiency of weight inheritance.

2. Methodology: The Three Dimensions of Slimming

SlimQwen's compression isn't just about deleting layers; it’s a surgical operation across three dimensions:

A. Depth & Width Pruning

The researchers found that dropping the last 25% of layers (Depth) and reducing the hidden dimension (Width) based on activation importance scores provides the best balance.

B. Expert Merging (The Secret Sauce)

MoEs offer a unique "expert" dimension. Simply deleting experts loses knowledge; merging them all causes "representation homogenization" (where every expert starts looking the same).

• The Solution: A Partial-Preservation strategy. Keep the top 50% of vital experts untouched to maintain specialization. Merge the remaining 50% into these bases to consolidate supplementary knowledge without muddying the waters.

C. Multi-Token Prediction (MTP) Distillation

Instead of just distilling the next token, SlimQwen distills the ability to predict multiple future tokens. This fundamentally enriches the student's hidden states and provides a massive boost to speculative decoding speeds, as the model becomes much more accurate at predicting sequences that a verifier will accept.

Figure: Overview of the SlimQwen pipeline, showcasing the transition from structured pruning to MTP-guided distillation.

3. The Power of "Progressive" Transitions

Perhaps the most insightful finding is that One-Shot compression is a trap. Suddenly yanking out 75% of a model's capacity shocks the optimization trajectory.

SlimQwen introduces Progressive Pruning:

1. Stage 1: Prune the model halfway (e.g., reduce only depth or only width) and train on a small token budget (40B tokens).
2. Stage 2: Complete the pruning to the final target and finish the training.

The Depth-first schedule emerged as the champion, yielding superior scores on MMLU (77.39 vs. 75.86 for one-stage). This suggests that letting the model adapt to a shallower structure before narrowing its width leads to more stable knowledge transfer.

4. Results: Efficiency Meets Intelligence

The resulting SlimQwen-23A2B is a powerhouse:

• Performance: Consistently beats random initialization across MMLU-Pro, GSM8K, and EvalPlus.
• Speed: Decoding throughput on vLLM jumped from 142.58 to 210.87 tokens/sec.
• Memory: Peak memory usage dropped from 156GB to 43GB, allowing a previously dual-GPU model to run comfortably on a single 80GB card.

5. Critical Insights & Future Outlook

While SlimQwen sets a new bar, it also highlights room for growth:

• The "Last Layer" Heuristic: The paper notes that simply pruning the last layers is often better than complex "importance-based" layer pruning, confirming a growing consensus that LLM depth redundancy is concentrated at the end of the stack.
• The Future of Distillation: The success of MTP KD suggests that future compression won't just focus on the "what" (next token) but the "logic" (sequence flow), making models inherently more optimized for inference-time speedups like speculative decoding.

In conclusion, SlimQwen proves that with the right schedule and a clever expert-merging strategy, we can "slim down" our largest MoE giants without losing their "brains."