Can world models learn failure dynamics from human interactions beyond successful demonstrations?

Two data gaps block action-conditioned world models

An action-conditioned world model takes a robot action sequence as input and predicts the resulting future states. That predictive ability is what makes such models useful for generating data, evaluating policies, and training in imagined environments. The paper argues that two data problems limit how well these models capture what actually happens during manipulation.

The first is coverage. Existing robotic datasets do not span the diverse interaction dynamics a generalizable world model needs. Large-scale egocentric human videos offer complementary interaction experience, but human hands and robot manipulators differ enough that transferring motion across that embodiment gap is difficult. Egocentric recordings also carry camera jitter and unstable viewpoints that can obscure the interaction region.

The second is outcome bias. Embodied datasets are dominated by successful demonstrations, so failure cases and off-target interactions are substantially underrepresented. A world model for embodied settings needs to anticipate erroneous or suboptimal actions, not only replay successful trajectories. The paper treats the absence of failure demonstrations as an open problem: how to synthesize failure scenarios that improve physical reasoning.

Stabilized pointflow links human hands to robot arms

HF-WM is a dual-branch framework that couples video generation with an auxiliary pointflow prediction branch. Pointflow here means tracked point trajectories: for each 81-frame video the method tracks 1008 points on a 24×42 grid, establishes their 3D reference positions from the first frame using depths predicted by Depth Anything 3, and reprojects them through subsequent frames using ground-truth camera poses.

That reprojection estimates motion caused by the camera rather than by the interaction. Subtracting it from the original CoWTracker trajectories yields stabilized point trajectories. The paper reports that static background points in raw pointflow drift coherently under egocentric camera motion, and that after compensation this global background motion is substantially reduced while hand and manipulated object motion is preserved. The authors read this as evidence that stabilization removes viewpoint-induced motion rather than smoothing all trajectories indiscriminately.

The pointflow branch is deliberately constrained. Its queries self-attend only within the branch and cross-attend exclusively to intermediate video features from the diffusion stream; skeleton, action, language, and other non-video keys and values are masked out. This prevents the branch from taking a shortcut through the conditioning signals and forces the pointflow objective to be grounded in the generated video. The paper also introduces a skeleton-grounded action representation that bridges human and robotic action spaces through a shared spatial formulation.

GRPO post-training turns perturbed successes into failure training

The failure side of HF-WM does not rely on real failure footage. Instead, the authors derive perturbed actions from successful demonstrations and post-train with Group Relative Policy Optimization (GRPO), guided by two rewards: action alignment and physical plausibility. The stated aim is to let the model learn plausible failure dynamics without paired ground-truth videos.

Training clips are anchored to gripper-state transitions. For each episode the method finds the frame with the largest absolute change in gripper state, then samples a 41-frame clip with stride 2 spanning 81 source frames, starting 12 source frames before the detected transition. A length-3 median filter suppresses transient fluctuations, and dual-threshold hysteresis classifies gripper states: values at or below 0.25 count as closed, values at or above 0.75 count as open, and intermediate values do not trigger a transition.

This yields a seven-way classification of windows, including close_hold, close_reopen, open_hold, open_close, remain_open, remain_closed, and partial_hold. Windows are sampled by class quota, with 75% allocated to the four classes that contain explicit gripper-state transitions and 25% to the remaining three. Within each window, the unperturbed route is chosen with probability 50%, keeping the original actions and video; otherwise the counterfactual perturbation branch runs. For windows with explicit transitions, gripper and geometric perturbations are sampled at 70% and 30%, giving within-class routing probabilities of 50% for no perturbation, 35% for gripper perturbations, and 15% for geometric perturbations. The paper states this balanced sampling is meant to prevent overemphasis on perturbed executions while the model still learns original executions.

A tiered checklist separates failure from physical violation

Because perturbed actions have no ground-truth video, the post-training reward needs a way to judge whether a generated rollout is physically acceptable. The paper describes a checklist shared by all candidates in a group. Tier-1 contains three critical checks aimed at severe physical or structural violations, such as object interpenetration, unsupported floating, and major inconsistencies in robot geometry. Tier-2 contains five fine-grained checks across physical consistency, robot structure, contact causality, and visual quality.

A separate model call answers each check with yes or no based on consecutive frames of the candidate video, and the physical plausibility score is the fraction of the eight checks passed. Any Tier-1 failure triggers hard rejection regardless of the Tier-2 score, so strong performance on other dimensions cannot compensate for a severe violation. The checks are tailored to scene-specific objects and counterfactual interaction conditions rather than requiring the original demonstration outcome to be reproduced; task failure alone is therefore not treated as a physical violation. This distinction matters, because the point of the exercise is to model unsuccessful executions plausibly, not to penalize them.

State-of-the-art generation and a policy-evaluation use case

The paper reports that HF-WM achieves state-of-the-art performance in action-conditioned video generation and shows strong utility for policy evaluation in downstream robotic tasks. The evidence packet supports these as the paper's headline claims; it does not supply the numeric margins, benchmark names, or per-task scores behind them, so the size of the improvement over specific baselines cannot be stated here.

The policy-evaluation use case is illustrated qualitatively. In one task, uncovering a wooden block, 1 of 6 policies succeeds on the real robot; in another, putting food on a plate, 5 of 6 succeed. Each strip compares a world-model rollout with real-robot video across six time steps, with the real column taken from the RoboArena human success label and the world-model column being the model's verdict. Disagreements are marked in red. These two tasks show the evaluation setup in operation rather than establishing an aggregate accuracy figure.

The failure claim rests on synthetic perturbations

The central limitation follows directly from the design: HF-WM never sees a real failure video during failure-aware post-training. Its failure dynamics come from perturbing successful actions and scoring the results against a checklist-based reward. The paper's own framing is that this enables learning plausible failure dynamics without paired ground-truth videos, which is a claim about plausibility under the model's reward, not a demonstration that the model reproduces recorded real-world failures.

The human-to-robot transfer carries its own boundary. Stabilized pointflow is presented as a unified motion representation across human and robot interactions, and the stabilization is shown to suppress background drift while preserving hand-object motion. But the paper's evidence for this is visualization on egocentric interactions, and the method depends on predicted depth and ground-truth camera poses for the compensation step.

The policy-evaluation evidence is likewise bounded. The two illustrated tasks differ sharply in real-robot success rate, 1 of 6 versus 5 of 6, and the packet reports per-task agreement patterns rather than a pooled correlation between world-model verdicts and real outcomes. Readers should treat the evaluation result as a demonstrated use case on these tasks, not as a calibrated predictor across the broader policy space.

Failure coverage, not just success coverage, is the lever

The paper's most transferable priority is the pairing it proposes: use egocentric human interaction video to widen interaction coverage, and use perturbed successful actions to widen outcome coverage. Neither ingredient alone addresses both gaps. Human video adds diverse interaction dynamics but not failures; successful robot demonstrations add real manipulation but remain success-biased. HF-WM's contribution is a concrete way to combine them, with stabilized pointflow handling the embodiment and camera-motion problem and GRPO post-training handling the missing-failure problem.

For readers judging this line of work, the decisive question is not whether the model can generate a failed-looking rollout, but whether its failure verdicts track real policy outcomes on tasks where success rates differ as much as 1 of 6 and 5 of 6. The paper reports state-of-the-art generation and useful policy evaluation; the next thing to watch is how well checklist-rewarded failure plausibility transfers to recorded real failures, which this study does not test.

Sources used in this answer