[CoRL 2024] TiPToP: The Return of Modular Planning in the Age of VLA

TL;DR

TiPToP (TiPToP is a Planner That just works on Pixels) is a modular robotic system that proves you don't always need 350 hours of demonstration data to "prime" a robot. By stitching together high-performance "foundation models" (for vision and semantics) with a high-speed GPU-parallelized planner (cuTAMP), TiPToP solves complex, multi-step manipulation tasks zero-shot. It outperforms state-of-the-art end-to-end models like π0.5-DROID in tasks requiring semantic reasoning and obstacle avoidance.

Problem & Motivation: The Data Wall vs. The Rigidity Trap

The robotics field currently faces a dichotomy:

1. VLA Models (e.g., π0.5, OpenVLA): These "End-to-End" models are easy to specify (Pixels in, Actions out) but are "data-hungry" and often have "black-box" failure modes. They struggle with long-horizon logic and distractor objects.
2. Classic TAMP: These systems are logically rigorous but "brittle." They usually require perfect 3D CAD models of every object and specific environment tuning, making them impossible to deploy in a standard home or office "out of the box."

TiPToP bridges this gap. The authors' insight is that Foundation Models have reached a tipping point where they can provide the "structured input" TAMP needs—meshes, grasps, and logical goals—directly from raw pixels.

Methodology: The Three Pillars of TiPToP

TiPToP decomposes the problem into a clear pipeline that allows for component-level debugging.

1. Perception: From Pixels to Logic

Instead of just predicting a "next action," TiPToP builds a world model.

• 3D Branch: Uses FoundationStereo for dense depth and M2T2 for 6-DoF grasp candidates.
• Semantic Branch: Uses Gemini 1.5 Pro to parse a command like "Serve the peanut butter crackers" into a logical goal: On(crackers, tray). SAM-2 then segmentizes the objects to create 3D meshes.

Fig 1. The TiPToP architecture: Modular perception feeds a GPU-accelerated planner.

2. Planning: cuTAMP

Traditional TAMP is slow. TiPToP uses cuTAMP, which parallelizes the search for trajectories across thousands of "particles" on a GPU. This allows the robot to "think" through multiple steps—like moving a soda can out of the way to reach a hidden snack—in just a few seconds.

3. Execution: Precision Control

Because the system is open-loop (it plans once and then acts), execution must be perfect. The team implemented a custom Joint Impedance Controller to ensure the robot follows the planned "timed trajectory" with sub-centimeter accuracy.

Experiments: David vs. Goliath

The authors compared TiPToP (Zero data) against π0.5-DROID, a state-of-the-art VLA fine-tuned on 350 hours of specific robot data.

The "Semantic" Advantage

VLAs often fail when given complex instructions like "Pick up the largest toy." They rely on statistical patterns. TiPToP, using an LLM backbone for grounding, identifies the specific object and sets a symbolic goal.

Table 1: TiPToP achieves significantly higher success rates in Distractor and Semantic categories.

Results Summary:

• Success Rate: TiPToP matched or beat the VLA in nearly all categories.
• Efficiency: TiPToP finished tasks roughly twice as fast as the VLA, which often "idled" or spent time on failed grasp attempts.
• Generalization: It was deployed on a UR5e and a WidowX AI arm in just a few hours—no retraining required.

Critical Analysis: Where does it break?

A major contribution of this paper is the Failure Analysis. By being modular, the authors can point to exactly which gear in the machine broke.

Fig 2. The Bottleneck: 31 out of 55 failures were due to Grasping.

The Main Catch: TiPToP is currently open-loop. If the robot misses a grasp or an object slips, it doesn't "see" the failure and continues moving as if it were holding the item. This is where reactive VLA models actually have an edge—they can fluctuate and "retry" mid-motion.

Conclusion: The Modular Future

TiPToP proves that we haven't "solved" robotics by just throwing more data at Transformers. Structured planning remains a superpower for long-horizon tasks and semantic reasoning. The future likely lies in a Hybrid Approach: using TiPToP to plan the "big picture" and using reactive VLA-style "skills" to handle the messy, high-frequency work of actually closing a gripper on a slippery object.

TiPToP is open-source, and its ability to be installed in under an hour makes it a potent base for the next generation of "out-of-the-box" robotics research.