[CVPR 2024] See, Plan, Rewind: Teaching Robots to Measure Progress and "Step Back" from Failure
See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation
The paper introduces See, Plan, Rewind (SPR), a progress-aware vision-language-action (VLA) framework that decomposes robotic tasks into explicit 2D spatial subgoals and semantic milestones. By monitoring these milestones, SPR enables autonomous error recovery through a "Rewind" mechanism, achieving SOTA outcomes on the LIBERO benchmark and superior out-of-distribution (OOD) robustness.
TL;DR
Robots often fail because they don't know they are stuck. See, Plan, Rewind (SPR) is a new VLA framework that gives robots "progress awareness" by breaking tasks into measurable 2D spatial subgoals. When the robot detects it’s not making progress—like a failed grasp or a collision—it autonomously "rewinds" to a safe state and tries again. It beats benchmarks like LIBERO by 5% and sets a new SOTA for out-of-distribution robustness.
The Motivation: Why Robots Need a "Rewind" Button
Traditional Vision-Language-Action (VLA) models are often "all-or-nothing." They take an image and an instruction, then output a stream of actions. But if a gripper slips or an object moves unexpectedly, these models lack the internal logic to realize the task has stalled.
Previous attempts at progress monitoring used abstract language (e.g., "Step 1 complete"), which lacks the spatial precision a robot needs to move. Other failure recovery methods require massive datasets of actual failures, which are expensive and dangerous to collect.
SPR's Insight: We can use successful demonstrations to define "spatial milestones." If the robot doesn't reach these milestones within a predicted timeframe, it should trigger a learned retraction—a "Rewind"—to reset itself for a better attempt.
Methodology: The See-Plan-Rewind Cycle
SPR operates through three distinct phases integrated into a single transformer-based VLA model (initialized from MolmoAct):
- See (Progress Grounding): The model predicts the remaining subtask count and the exact 2D coordinates for each transition (e.g., the moment a gripper should close on a mug).
- Plan (Trajectory Grounding): Instead of planning for the final goal 100 steps away, SPR plans a 2D trajectory of waypoints to the next immediate subtask goal.
- Rewind (Anomaly Recovery): A state recorder tracks subtask counts and trajectories. If the count increases (regression) or the trajectory stays identical for too long (stagnation), the model switches the instruction to "return to initial position" for a brief period before resuming.
Figure 1: The SPR framework workflow showing the closed-loop execution and the trigger-based Rewind mechanism.
Automated Data Curation
To avoid manual labeling, the authors used:
- Gripper States: Detecting open/close events to segment subtasks.
- DINOv3 + SAM: Automatically extracting 2D gripper coordinates throughout the videos.
- DeepSeek-R1 / Gemini: Generating semantic descriptions for each identified subtask segment.
Experimental Results: Robustness Under Pressure
1. SOTA on LIBERO-Plus (OOD Generalization)
The true test for any robotic policy is how it handles "unseen" variations (different lighting, new layouts, varying instructions). SPR showed incredible resilience on LIBERO-Plus, maintaining the highest success rates with the lowest performance drop compared to heavyweights like OpenVLA and UniVLA.
| Method | Background | Robot Pose | Language Vari. | Average Drop |
|---|---|---|---|---|
| OpenVLA-OFT | 83.6% | 30.6% | 83.6% | -27.0% |
| SPR (Ours) | 86.0% | 47.7% | 78.5% | -18.8% |
2. Real-Robot Performance
In the "Tidy up the Table" task (long-horizon sorting of 4 objects), the baseline MolmoAct failed completely (0%). SPR achieved a 30% success rate—proving that subtask decomposition is the only way to scale to complex, multi-step real-world scenarios.
Figure 2: Performance improvement over extended episode lengths. SPR continues to improve after baselines plateau, thanks to its ability to retry failed subtasks.
Critical Analysis & Future Outlook
Why it works
The "Rewind" mechanism effectively moves the robot back to an "in-distribution" state. Failures often push the robot into weird poses that the model never saw during training (Out-of-Distribution). By retreating to the start, SPR returns the robot to a visual state it recognizes, allowing it to "try again" with a clean slate.
Limitations
- Physical Constraints: As noted in the failure analysis, if a robot is physically "stuck" (wedged in a microwave), a software-level rewind command might not generate enough torque to overcome physical friction.
- Precision: Action tokens are still discrete, leading to occasional "near-misses" in high-precision tasks like inserting a peg.
Conclusion
SPR moves away from the "black-box" nature of end-to-end VLAs by re-introducing classical concepts of subgoals and milestones, but doing so within a modern, differentiable transformer architecture. It proves that the path to robust robotics isn't just "more data," but smarter, progress-aware reasoning.