[ICLR 2025] ICLR: Elevating Robotic In-Context Learning with Embodied Visual Reasoning
ICLR: In-Context Imitation Learning with Visual Reasoning
ICLR (In-Context imitation Learning with visual Reasoning) is a novel framework that enables robots to adapt to unseen tasks using a small number of demonstrations without fine-tuning. It introduces structured visual reasoning traces—represented as predicted future gripper trajectories in image space—integrated into a unified autoregressive transformer to achieve SOTA performance in complex manipulation tasks.
TL;DR
Adapting robots to new tasks with just a few demonstrations (In-Context Learning) often fails because models "mimic" movements without "understanding" the intent. ICLR (In-Context imitation Learning with visual Reasoning) fixes this by forcing the model to visualize its future trajectory in the image space before acting. By treating reasoning as a sequence of tokens in a Transformer, it achieves massive performance gains in both simulation (+230%) and real-world robotics.
Problem & Motivation: The "Blind Mimicry" Trap
Current in-context imitation learning (ICIL) models like ICRT treat robotics as a pure sequence-to-sequence problem: [Demo States/Actions] -> [Target State] -> [Predict Action].
However, the authors point out a fundamental flaw: Ambiguity. In a kitchen with a lion toy, a dumpling, and a box, simply moving the hand toward the center of the table could mean "pick up the lion" or "move the dumpling." Without explicit task intent, the model often gets confused by distractors.
The researchers' insight was to borrow from the LLM world's "Chain-of-Thought" (CoT) prompting. Instead of jumping straight to the action, the robot should first "think" about where it plans to go in the visual field.
Methodology: Reasoning as a Modality
ICLR redefines the input-output stream of the robot. Instead of just States () and Actions (), it introduces Reasoning Traces ().
1. Generating the "Thoughts"
The reasoning trace is a polyline of 5 keypoints in the image space tracking the gripper's future positions.
- In Simulation: Derived from ground-truth 3D poses.
- In Real World: Generated using the Molmo2 VLM, which can "point" to the gripper in images with high precision.
2. The Unified Transformer
The architecture uses a Llama2-style causal transformer. It interleaves tokens from different encoders:
- State Encoder: Multi-view RGB + Proprioception.
- Reasoning Encoder: Pixel-space polylines.
- Action Encoder: Delta end-effector poses.
Fig 2: ICLR architecture showing the interleaving of state, reasoning, and action tokens.
The model is trained via next-token prediction. Crucially, it must predict the next reasoning trace before the next action chunk.
3. Reasoning Dropout
To prevent the model from becoming a "slave" to its own (potentially noisy) predicted paths, the authors introduced Reasoning Dropout—randomly masking reasoning tokens during training. This forces the model to be robust even if the visual reasoning is slightly off.
Experiments & Results: A New Baseline for ICIL
The framework was tested on the LIBERO simulation suite and a real-world Franka Panda setup with a UMI gripper.
Simulation: Dominating LIBERO
On 100+ tasks, ICLR showed a dramatic lead. Interestingly, in simulation, "Ours Dropout" (using reasoning during prompts but skipping it during target inference) performed best, likely due to the lower visual diversity of simulation which allows the model to "internalize" the logic.
Table I: ICLR outperforms the ICRT baseline by over 3x in average success rate.
Real-World: Handling Unseen Objects
In the real world, the Full Reasoning model (generating traces during inference) was the clear winner. This proves that in complex, high-entropy environments, explicit reasoning is non-negotiable. It successfully handled completely unseen objects like a "toy monkey" and "potato" by observing a single prompt demo.
Deep Insight: Interpretable Failures
One of the coolest features of ICLR is its transparency. Because the model outputs a visual trace, we can see why it fails.
- If the trace points to the wrong object, the Reasoning failed.
- If the trace is correct but it misses the grasp, the Control failed.
The authors found that 40-45% of failures were due to faulty reasoning traces, while the rest were low-level execution errors. This tells the community that while reasoning helps, we still need more robust low-level controllers.
Conclusion & Future Outlook
ICLR proves that "Action-Only" imitation is hitting a ceiling. By integrating Visual Reasoning, we give robots a way to represent "intent" without needing complex language descriptions for every sub-step.
Future Work: The authors suggest expanding to bimanual manipulation and using human videos as prompts. As VLMs like Molmo and GPT-4o become faster, real-time visual CoT will likely become a standard component of the "Robot Brain."
Disclaimer: This post is a technical breakdown of "In-Context Imitation Learning with Visual Reasoning" (2025/2026).