What causes long-horizon skill-seam failures in robotic manipulation?

A downstream skill inherits an unfamiliar scene

Long-horizon manipulation is often assembled by training skills separately and running them in sequence. Each skill can look reliable when tested from its own starting states, yet the chain collapses: on the three-skill chains studied in BOSS-44, success falls from 88.5% after the first skill to 34.3% after two skills and 7.6% after all three, even though the component skills average 79.0% success when evaluated independently from their canonical initial states.

The paper names this mismatch Observation-Space Shift (OSS). At a skill seam, the next skill does not begin from the states it was trained on; it begins from whatever its predecessor left behind. An "open drawer" skill may leave the drawer open before a "place bowl" skill trained with the drawer closed starts executing. The downstream policy is then asked to act from a state outside its support, and it can stall even though it succeeds in isolation.

Resetting the arm alone changes almost nothing

To find what actually causes the stall, the authors use a privileged teleportation oracle in simulation and restore one scene component at a time to its canonical value at the moment the monitor fires. The comparison is run on 100 catastrophic chains, and the metric is success through the full chain.

The base condition reaches 0.098. Restoring only the robot arm gives 0.132, within noise of the base, which rules out the idea that the arm's own configuration is the bottleneck. Restoring the arm plus drawer articulation gives 0.212, and arm plus secondary objects gives 0.268. Restoring everything, the oracle, reaches 0.518. The object the next skill manipulates is usually already close to canonical: median displacement is 1 cm, and it sits 25 cm from the gripper. Almost nothing is physically lost, with 2 of 10,036 states dropped, or 0.02%.

Drawer state and leftover objects must be fixed together

The two dominant components do not simply add up. Restoring articulation or secondary objects individually yields only partial gains, and the individual increments sum to a predicted 0.35, while joint restoration yields 0.52, an excess of 0.170 with propagated standard errors of 0.052, roughly 3.3 sigma. The authors read this as complementary effects: partial fixes do not stack, and the scene has to be restored jointly.

That decomposition is a coarse diagnostic run with its own base of 0.098, which differs from the 0.076 five-seed base used for the headline result. The paper is explicit that the component table is used to localize the cause, not to report the main success number.

More sampling, a stronger policy, and world models all fail

If seam collapse were bad luck, a weak base policy, or a planning failure, a cheaper intervention than restoration should work. The paper tests each in its strongest form from the actual seam states, and none recovers.

Best-of-K resampling with K=8 succeeds from a clean initialization, where at least one rollout works and selection is trivially perfect, but from the mid-chain seam the base policy fails on all eight rollouts, leaving nothing to select. A Diffusion Policy trained for 2,500 epochs on the same canonical demonstrations matches behavior cloning exactly from seam states, at 0.00, even though it succeeds from canonical initial states. A world model fails in all three roles tested: as a frame generator it ties the baseline, as a monitor its prediction surprise inverts on stalls, and as a planner model-free RLPD exceeds the model-based alternatives at three times the sample efficiency. End-to-end RL recovery stays below the demonstration baseline, at 0.45 versus 0.52.

Restoring the scene before resuming drives the gain

The deployed system has three learned parts. A task-progress monitor built on a frozen DINOv3 encoder and a small task-conditioned head flags the stall. A learned restoration policy, trained per scene with SAC or demonstration-augmented RLPD, returns the displaced drawer and objects toward their targets. Seam-robust fine-tuning then lets the skill resume, using 581 deep hand-off demonstrations mixed with the original per-skill data.

On the 10 catastrophic chains over five seeds, the full system raises full-chain success from 0.076 to 0.265, a gain of 0.189 with p=0.047. A base given the same re-home and retry budget reaches only 0.129, so the effect is not just extra execution steps. Neither component alone is significant: recovery only reaches 0.175 at p=0.094, and prevention only reaches 0.114 at p=0.188. The claim is therefore at the system level, and the system reaches 51% of the privileged oracle's 0.518, leaving roughly half the recoverable gap to restoration quality rather than detection.

Restoration helps when a drawer blocks a non-drawer skill

The per-chain analysis isolates when the restore action pays off. The restorer acts only on the drawer, so the outcome tracks one variable: whether the seam's blocking displacement is a drawer that the downstream skill does not itself operate. An off-canonical drawer blocking a non-drawer skill is the case restoration addresses. Chains whose second seam displaces a free object stay unrecovered and would need an object-restoration skill.

The scene split is uneven. The full system reaches full-chain success of 0.29 on S1, 0.34 on S2, and 0.16 on S4, where the base is 0.19; one S4 chain regresses from 0.27 to 0.17 under the full system, a false-alarm cost the authors report rather than average away. On seven held-out reorderings of the same skills, producing seam transitions never used to design the system, success improves from 0.066 to 0.244, a 3.7x gain that is too small alone to reach significance at p=0.125, though the oracle corroborates the direction at 0.066 to 0.499.

On a real arm, the monitor is limited by what the camera sees

The simulation monitor runs unchanged on a Franka arm driven by a served, LoRA-fine-tuned pi-0.5 vision-language-action policy across three rearrangement chains, nine skills, and 15 trials per chain. Detection weakens sharply. Scored leave-one-episode-out on known objects, AUROC is 0.625 with a 95% interval of 0.46 to 0.79; leave-one-chain-out on unseen objects it is 0.672 with an interval of 0.51 to 0.83. At n=45 neither separates from chance, and the paper reports weak transfer rather than a significant detection claim.

The gap is observability: with an exterior camera alone, a failed grasp or dropped object is often invisible even when the place motion still completes on camera. Closing the loop is still a proof of concept. At matched detector flags, restoration recovers 3 of 21 otherwise-terminal place attempts while the baseline recovers 0 of 19, an effect of the restore action that does not depend on detector quality because both conditions use the same live flags. Per-skill success is otherwise within noise at N=15, and the authors claim no success-rate improvement on hardware.

The remaining gap is restoration quality, not detection

The paper's own framing is that seam failure is an off-support state rather than a miscalibrated policy, which is why searching harder, swapping the policy, or planning all leave the seam unmoved. The ablation says what to restore: articulation and objects are complementary, and any single-component fix reaches only 0.13 to 0.27 against the 0.52 oracle.

The priority the evidence supports is therefore not more test-time compute or a bigger base policy, but better restoration and better sensing. Even where restoration applies it recovers only about half the oracle gap, and the object restorer does not yet transfer. The authors hypothesize that re-entry timing, when control is handed back, is the limiting factor rather than component coverage, since the monitor can re-fire mid-restoration and a fixed hand-back tolerance is a crude proxy for "restored enough to resume." On hardware, the concrete fix they name is wrist and gripper sensing, because gripper width and proprioception were not logged and a failed grasp is invisible to an exterior camera. The practical split is thus: restore the scene when a displaced drawer blocks a skill that does not operate it, and invest in sensing and re-entry timing before expecting the loop to carry a real robot.

Sources used in this answer