[E3-TIR] Beyond Imitation: Boosting Tool-Integrated Reasoning via Enhanced Experience Exploitation

TL;DR

Training LLMs to use tools (like code interpreters or search engines) effectively is notoriously difficult. Standard Reinforcement Learning (RL) starts too slow, while Supervised Fine-Tuning (SFT) makes models rigid. E3-TIR bridges this gap by using a "Warm-up" phase that branches out from expert prefixes. It achieves a 6% performance boost with 1/10th of the data, effectively solving the "react mode" degradation where models forget to think and only act.

The "React Mode" Trap and Capability Rigidity

The authors identify two fatal flaws in current agent training:

1. Zero-RL's Inefficiency: Without guidance, models stumble into "React Mode"—invoking tools blindly without intermediate Chain-of-Thought (CoT) reasoning. This leads to high failure rates and slow convergence.
2. SFT-then-RL's Collapse: When a model is over-trained on static expert data (SFT), it loses the "entropy" needed for RL. During the RL phase, it often undergoes low-entropy collapse, where it converges to simplistic, repetitive patterns and fails to explore the vast space of complex reasoning.

Figure 1: Comparison of Zero-RL (slow), SFT+RL (unstable), and E3-TIR (stable & fast).

Methodology: The Three Stages of E3-TIR

E3-TIR reframes the training as a dynamic integration of three experience types:

1. Prefix-Guided Branch Sampling

Instead of starting from zero or just copying an expert, E3-TIR identifies high-entropy anchors in expert trajectories. These are the "forks in the road" where the model is most uncertain. By branching out from these anchors, the model explores diverse reasoning paths while staying within the "neighborhood" of a correct solution. Mathematical Intuition: This reduces the "Curse of Horizon." If a task takes 8 steps, a random success is $(0.4{)}^8\approx 0.0006$. By anchoring at step 6, the success probability for the remaining 2 steps jumps to $(0.4{)}^2=0.16$—a massive improvement in signal density.

2. Advantage-Aware Gradient Detaching (AAGD)

A major challenge in tree-based sampling is that a single failed branch can "poison" the shared prefix during backpropagation. E3-TIR uses AAGD to selectively block gradients from failed exploration paths (Advantage < 0) while allowing successful ones to update the prefix. This ensures the foundational reasoning steps (the "trunk" of the tree) are not degraded by experimental failures in the "leaves."

3. Hybrid Advantage Estimation

The framework calculates two types of rewards:

• Global Advantage: How good is this path compared to everything we've seen?
• Expert-Tree Internal Advantage: How does this branch compare to other branches starting from the same expert anchor?

Figure 3: Detailed architecture of the E3-TIR branching and filtering mechanism.

Experimental Showdown

The results across Mathematical (AIME, MATH500) and Knowledge-Intensive (HotpotQA, Musique) tasks are striking.

• Efficiency: E3-TIR achieves its gains using less than 10% of the synthetic data used by standard SFT models.
• Robustness: The code execution failure rate dropped to 4.0%, nearly half that of Zero-RL (7.4%).
• ROI: The framework yields a 1.46x ROI gain, proving it is more computationally efficient for reaching a desired capability level.

Table 1: Benchmark results across different model scales (3B, 7B, 8B).

Critical Insight: The "Solve-None" Barrier

One of the most profound metrics tracked is the Solve-None rate—the proportion of problems the model simply cannot solve regardless of the number of tool calls. E3-TIR shows a continuous decline in this rate, suggesting that active exploration around expert anchors actually expands the capability boundary of the agent, rather than just optimizing its current knowledge.

Conclusion

E3-TIR shifts the paradigm from "passive imitation" to "proactive acquisition." By treating expert data as a starting point for exploration rather than an absolute target, it overcomes the rigidity of SFT and the inefficiency of pure RL. For developers of LLM agents, the lesson is clear: don't just fine-tune on trajectories; branch off of them.