[ICRL] Training LLMs for Tool Use Without SFT: A Curriculum-Based RL Approach
In-Context Reinforcement Learning for Tool Use in Large Language Models
In-Context Reinforcement Learning (ICRL) is an RL-only framework designed to train Large Language Models (LLMs) for complex tool use (e.g., search engines, code execution) without the need for supervised fine-tuning (SFT). By integrating few-shot prompting during RL rollouts and utilizing a curriculum-based reduction of these prompts, ICRL achieves new SOTA performance on reasoning benchmarks like TriviaQA and Musique.
TL;DR
Building models that can use tools (like search engines or Python) usually requires a massive amount of human-annotated "tool-use traces" for Supervised Fine-Tuning (SFT). In-Context Reinforcement Learning (ICRL) flips the script: it skips SFT entirely. By using few-shot prompts during the RL exploration phase and gradually removing them, ICRL teaches models to reason and call tools autonomously, achieving SOTA results with significantly higher data efficiency.
Motivation: The High Cost of the "Cold-Start"
The current dominant paradigm for training "Reasoning Models" (like DeepSeek-R1 or Search-o1) involves a two-stage process:
- SFT (Cold-Start): Fine-tune the model on structured trajectories so it learns the format (e.g.,
<think>,<search>). - RL (Alignment/Reasoning): Optimize for accuracy and efficiency.
The bottleneck is the SFT data. Annotating multi-hop search queries or complex code-execution paths is incredibly expensive. However, if we skip SFT, the model "wanders" in the action space, failing to generate the correct XML tags or logic, leading to a total failure in RL exploration. No valid tool calls means no reward, and no reward means no learning.
Methodology: Soft Supervision via In-Context Learning
The core insight of ICRL is that In-Context Learning (ICL) can serve as a bridge. Instead of baking the "how-to" into the model's weights via SFT, the authors put the "how-to" in the rollout prompt.
1. The Rollout Template
During RL training, when the model generates trajectories, it sees a prompt containing examples of perfect tool-use reasoning. These examples guide the model's exploration without updating weights through supervised gradients.
2. The Multi-Stage Curriculum
To ensure the model doesn't become a "prompt-dependent" agent, ICRL introduces a curriculum:
- Stage 1 (Imitation): Rollouts use 3-shot examples. The model learns the format and basic logic.
- Stage 2 (Transition): Examples are reduced to 2-shot. The model relies more on its updated weights.
- Stage 3 (Autonomy): 0-shot. The model must generate tool calls and reasoning steps entirely on its own.
3. Masked GRPO
The authors use Group Relative Policy Optimization (GRPO). A critical technical detail here is Loss Masking: the tokens returned by the tool (e.g., the search result text) are masked out. This ensures the RL objective only optimizes the model's decision-making (when to search, what to query, how to answer) and not the external content it retrieves.
Experiments: Breaking the SOTA
The results are particularly striking in multi-hop reasoning (tasks where you need to search for A to find B, then search for B to find the answer).
Superior Performance on 3B and 7B Models
ICRL achieved an average Exact Match (EM) score of 49.12 on Qwen2.5-7B, outperforming ParallelSearch (41.78) and Search-R1 (38.16).
Table: Comparison across benchmarks like TriviaQA, HotpotQA, and Musique.
The SFT Mystery
The most impressive finding is in Table 4 of the paper: ICRL (No SFT) actually outperforms O2-Searcher (With SFT). On TriviaQA, the gap is a massive 12.9%. This suggests that "learning through doing" with ICL guidance is more effective for reasoning than "learning through memorizing" via SFT traces.
Deep Insight: Why Curb the Curriculum?
The authors performed an ablation study on the curriculum stages. They found that a "3-2-0" stage transition performed much better than a "3-2-1-0" transition.
- The Paradox: Reducing the prompt too slowly (adding a 1-shot stage) actually made the model lazier, causing it to terminate searches too early.
- The Takeaway: A sharper transition to zero-shot forces the model to "step up" and internalize the long-range reasoning dependencies seen in the 2-shot/3-shot examples.
Figure: The evolution of response length and rewards across curriculum stages.
Conclusion and Future Outlook
ICRL proves that we don't need millions of dollars in SFT data to create world-class tool-using agents. By smartly combining the inductive bias of prompts with the optimization power of RL, we can bootstrap complex behaviors from scratch.
Limitations
- Prompt Length: During early training stages, the prompts are long (carrying examples), which increases the compute cost for rollouts.
- Prompt Sensitivity: The performance might still depend on the quality of the few-shot examples provided in the initial stage.
Future work could look into Self-Iterative ICRL, where the model's own successful zero-shot trajectories are used as the new "in-context examples" for the next generation of training, creating a truly autonomous fly-wheel of intelligence.