[ICLR 2025] MAGE: Internalizing Multi-Agent Strategy in LLMs via Meta-RL

TL;DR

Current LLM agents are often "static task-solvers" that struggle to adapt when environments shift or when facing strategic opponents. MAGE (Meta-RL for Language Agents) transforms these agents into "strategic learners." By using a multi-episode training objective focused on final-episode success and population-based training, MAGE allows agents to internalize the ability to probe, reflect, and exploit opponent vulnerabilities in a zero-shot manner.

Problem & Motivation: The Gap in Strategic Adaptation

While LLMs can use In-Context Learning (ICL) to follow prompts, they rarely evolve their strategy over multiple interactions without external help. Standard Reinforcement Learning (RL) typically optimizes for a fixed policy, while existing Meta-RL for LLMs (like LAMER) focuses almost exclusively on exploration in single-agent tasks.

In the real world—and specifically in multi-agent games like Poker or Tic-Tac-Toe—an agent must perform strategic exploitation. This means identifying the specific "tell" or weakness of a particular opponent and changing tactics mid-stream. The authors argue that this "learning-to-learn" logic must be internalized within the model's weights, not just added as a prompt.

Methodology: Learning to Learn via Reflection

MAGE treats a sequence of $N$ episodes as a meta-episode. The core innovation lies in its "Reflective Inner Loop":

1. Multi-Episode Integration: Interaction history and self-generated reflections ($m_{n-1}$) from previous trials are fed into the context window.
2. Differential Meta-Reward: Instead of maximizing cumulative reward, MAGE optimizes the Final Episode Reward. Specifically, it uses a differential reward $R_n-R_{n-1}$ to measure the improvement gained from reflection.
3. Strategic Multi-Agent Training: To prevent the model from over-fitting to one strategy, MAGE uses Population-Based Training (PBT). It trains against a pool of diverse opponents (e.g., Aggressive, Conservative, MCTS-based).
4. Agent-Specific Normalization: To handle the high variance in rewards when switching between easy and hard opponents, MAGE applies advantage normalization specific to the opponent type, ensuring stable gradient updates.

Experiments & Results: Beyond Pattern Memorization

The authors tested MAGE across single-agent (ALFWorld, WebShop) and multi-agent (Tic-Tac-Toe, Kuhn Poker) benchmarks.

• Human-Level Strategy: In Kuhn Poker, MAGE reached the theoretical game-theoretic ceiling against CFR (Counterfactual Regret Minimization) opponents.
• Rapid Adaptation: In WebShop, MAGE achieved a perfect 100% success rate, whereas standard RL baselines like GiGPO plateaued around 79.7%.
• Generalization: Most impressively, MAGE maintained high performance against unseen opponents (OOD). Against a powerful MCTS-1000 bot in Tic-Tac-Toe, MAGE's draw rate ascended to 100% as the episodes progressed, proving it learned the logic of defense rather than just memorizing moves.

Ablation Insight: Why Final-Episode Optimization?

Ablation studies showed that optimizing for the "Differential Return" (the progress between episodes) is the primary driver for the steep learning curve. Standard RL objectives often lead to "conservative" play, whereas MAGE's objective encourages "strategic probing" in early episodes to ensure late-episode victory.

Critical Analysis & Conclusion

MAGE provides a compelling case for Meta-RL as the preferred way to build adaptive agents. By moving the adaptation logic from the prompt (ICL) to the weights (Meta-RL), the agent becomes significantly more robust to non-stationary environments.

Takeaways:

• Strategic Plasticity: MAGE doesn't just solve tasks; it solves the process of learning the task.
• Visual Proof: 3D manifold visualizations of the state-action space show that MAGE creates "policy tunnels"—highly consistent and stabilized trajectories that emerge after a few episodes of exploration.

Limitations: Currently, MAGE is focused on discrete, text-based environments. The next frontier will be scaling this to high-dimensional, open-ended real-world interactions and multi-modal feedback loops.