FST: Breaking the Bottleneck of LLM Adaptation with Fast-Slow Training

TL;DR

Researchers have long struggled with a fundamental trade-off: updating model weights (RL/SFT) offers high performance but destroys the model's "plasticity" (the ability to learn new things), while prompt engineering is flexible but limited in power. Fast-Slow Training (FST) reconciles these by co-evolving a population of optimized prompts ("fast weights") and model parameters ("slow weights"). The result? Models that learn 3x faster, reach higher accuracy, and—most importantly—don't "forget" how to learn.

The Problem: The High Cost of Learning

Current LLM post-training is "one-size-fits-all." Whether an improvement is a permanent reasoning skill or a tiny task-specific hint, it is forced into the model's weights. This leads to three major issues:

• Catastrophic Forgetting: As the model aligns with Task A, it forgets general knowledge.
• Loss of Plasticity: The weights become so "brittle" that the model struggles to learn Task B later.
• Drift: The model moves too far from its original, high-entropy base state (as measured by KL Divergence).

Methodology: A Division of Labor

FST draws inspiration from the human brain's System 1 and System 2 (Fast vs. Slow). It instantiates this through two interleaved loops:

1. The Fast Loop (Context Optimization): Using an evolutionary method called GEPA, the system maintains a "population" of prompts. These prompts are updated based on rich textual feedback (critiques) from a reflection LM. This allows the model to "learn" task heuristics immediately without touching a single parameter.
2. The Slow Loop (Policy RL): Conventional RL updates the parameters $heta$. However, unlike standard RL, these updates happen under the context of the evolving prompts.

The Architecture of Co-evolution

By maintaining a Pareto-frontier of prompts, the system ensures diversity. Different prompts specialize in different "slices" of the task, providing the RL algorithm with a richer, more stable signal for gradient updates.

Experimental Breakthroughs

The researchers tested FST across Math (Polaris), Coding (CodeIO), and Multi-hop reasoning (HoVer-hard).

1. Superior Data Efficiency

FST isn't just better; it's faster. In reasoning tasks, FST matched the peak performance of standard RL with 3.0x fewer training samples. It effectively "escapes" the zero-reward regime of hard tasks much earlier because the fast textual weights inject task structure almost instantly.

2. Preserving the "Brain's" Flexibility

Perhaps the most striking result is the Plasticity Probe. When training on Task A (Math) and then Task B (HoVer), RL-only models completely collapsed on Task B. FST models, because their slow weights stayed "closer" to the base model (70% less KL drift), were able to master Task B with nearly the same efficiency as a fresh model.

3. Continual Learning Success

In a sequential setup (HoVer $\to$ CodeIO $\to$ Physics), standard RL stalled completely by the second task. FST maintained high performance across the entire stream, proving its viability for "evergreen" AI systems that must adapt on the fly.

Deep Insight: Why Does It Work?

FST works because it correctly identifies declarative vs. procedural knowledge.

• Fast weights (textual) should handle declarative task-specific heuristics (e.g., "don't forget to escape backslashes in JSON").
• Slow weights (parametric) should focus on procedural consolidation (general reasoning logic).

When we force the parameters to absorb everything, they over-fit to specific labels. By offloading the "chaff" to the prompt, the "wheat" (reasoning) is cleanly encoded in the weights.

Critical Analysis & Future Outlook

Limitations: FST currently relies on an external Reflection LM (like GPT-4) to evolve prompts, which adds API costs and latency. Additionally, while the "slow" side is more efficient, the hybrid training process is more complex to orchestrate.

Future Work: The true potential of FST lies in autonomous systems. Imagine a model that manages its own long-term memory (weights) and short-term working context (prompts) to survive in evolving environments without ever needing a "full retrain." This work provides the first robust mathematical and empirical framework for such a future.

Conclusion (Takeaway)

The future of LLM post-training is not just "bigger data" or "more RL." It is multi-channel optimization. By letting prompts and parameters co-evolve, FST builds models that are not only smarter but also more resilient and adaptable to the ever-changing landscape of real-world tasks.