Can Forward Dynamics Models Couple Action and Future Prediction Without Error Propagation?

Two paradigms each leave a gap

Vision-language-action (VLA) models adapt pretrained vision-language models to map the current observation and task instruction directly to actions. The paper's criticism is that this action-imitation route learns what to do without explicit supervision on what happens next, so the signal contained in future world states goes unused.

World action models (WAMs) build on video backbones and model visual dynamics alongside actions. In the prevalent design the model first predicts future world states and then infers actions from them in an inverse-dynamics manner. The paper argues that future prediction is inherently error-prone because every visual detail matters across space and time, so placing it before action prediction lets those errors accumulate and amplify into actions, and future generation adds substantial inference overhead.

FDM is the paper's attempt to keep the training benefit of future states while removing the inference dependence on predicted futures. The authors describe it as an action-to-future paradigm: future prediction is conditioned on action features, never the reverse.

Asymmetric attention sets the information direction

Both FDM instantiations connect action learning and future prediction through layer-wise attention between experts. Each expert keeps its own feature transformations, and attention lets it read only the representations it is permitted to see. The future-prediction branch reads action features throughout the network rather than the final numerical action output.

The attention mask is what enforces the direction. In the VLM-based model, context features are available to both downstream experts, and generation tokens additionally read action features. In the VGM-based model, current-observation tokens attend only to the current observation, action tokens read the current observation and other action tokens, and future-video tokens read all three groups. These permissions hold at every layer, which keeps action features and the observation features they read independent of generation-branch tokens.

The same connection carries supervision in the reverse direction during optimization. The future loss differentiates through the action keys and values read by the generation expert and updates the action features that produce them, while action imitation simultaneously supervises those features through the action velocity head. That is how ground-truth future states reach the policy without the policy ever consuming a predicted future.

Future supervision raises success across backbones

The paper instantiates FDM twice to test whether the benefit survives a change of backbone. FDM w/ VLM extends π0.5 with a randomly initialized generation expert of the same architecture as its action expert; FDM w/ VGM adds a randomly initialized action expert to Wan2.2-5B. Both are trained without robot policy pretraining.

On LIBERO, FDM w/ VLM raises average success from 96.9% for π0.5 to 99.4% and lifts the LIBERO-Long suite from 92.4% to 98.6%. FDM w/ VGM reaches 98.1% average success, against 97.7% for Motus and 98.5% for LingBot-VA. On RoboTwin, the two instantiations average 81.20% and 89.23%, improving on π0.5 (79.75%) and Motus (87.8%), and both improve under randomized conditions.

The real-robot evaluation uses a dual-arm ARX-X5 robot and five tasks: drawer packing, box packing and closing, shuttlecock insertion, cup stacking, and block stacking, with 20 trials per method per task. FDM w/ VLM averages 96% success against 83% for π0.5 on all five tasks. FDM w/ VGM reaches 83%, against 78% for the architecture-matched FastWAM-Joint and 71% for DreamZero. Both instantiations reach 100% on Cup and Block.

Backpropagation through action features carries the gain

A success-rate comparison alone cannot show which part of the design matters, so the authors ablate the action-to-future connection on the five real-robot tasks with four variants under matched training and evaluation settings and the same training seeds. Action-only trains with action imitation alone. Context-only future adds the future objective but blocks generation-branch tokens from reading action features. Stop-gradient action conditioning lets the future branch read sg(Ha), preserving forward values but blocking gradients. Full FDM lets the future branch read Ha and backpropagate through it.

Adding a context-only future objective raises average success from 83% to 87% for the VLM-based model and from 66% to 72% for the VGM-based model. Full FDM then improves over that control by 9 and 11 percentage points, with higher success on all five tasks in both families. Relative to action-only training, total gains reach 13 points for the VLM-based model and 17 points for the VGM-based model.

The cleanest isolation is Full FDM versus Stop-gradient. For identical parameters, inputs, and noise, sg(Ha) and Ha yield identical forward predictions, so the only difference is whether future-loss gradients pass through the action-feature interface during training. Full FDM raises average success from 89% to 96% for the VLM-based model and from 78% to 83% for the VGM-based model, gains of 7 and 5 points. In each family it improves four tasks and matches Stop-gradient on the fifth. The authors read this as evidence that backward supervision through action features, not merely the presence of a future objective, drives the deployed action-only policy's improvement.

Perturbing predicted futures leaves actions unchanged

The paper tests the error-propagation claim directly rather than inferring it from success rates. It takes FastWAM-Joint, which shares FDM w/ VGM's architecture and differs only in its attention mask, and injects Gaussian noise once into the future-video latent at an intermediate stage of joint denoising, leaving the current observation and noised action chunk unchanged. Perturbation strengths are α ∈ {0, 0.1, 0.25, 0.5, 1.0}, with α = 0 as the unperturbed condition, and the reported measure is the percentage increase in action error relative to that condition.

The evaluation is open-loop across RoboTwin, LIBERO, and the five real-world tasks, using the full validation set for each, with observation, instruction, and sampling randomness held fixed per sample. FastWAM-Joint's action error rises with perturbation strength. For FDM the change is zero by design: for fixed parameters, inputs, and action-sampling randomness, perturbing the generated future leaves action predictions unchanged regardless of future-generation quality.

This is an architectural invariance, not an empirical robustness result. It follows from the conditioning direction: FDM's action path never reads generation-branch tokens. The open-loop action-error measure is also a different quantity from closed-loop task success, and the paper reports them separately.

Future and action gradients point broadly together

If future supervision were fighting action imitation, the training signal would be suspect even if the architecture were clean. The authors compare action-imitation gradients with the future-loss contribution propagated through the action-to-future interface, measuring cosine similarity on shared-context key/value projections. This isolates the backward contribution through action features rather than using the total future-loss gradient.

FDM w/ VLM shows predominantly positive alignment across the evaluated settings, with only limited early-stage deviations. Most FDM w/ VGM settings also show positive alignment after initial fluctuations. The authors conclude that the two gradient directions are broadly compatible in this analysis and treat this as support for using the action-to-future connection as an auxiliary training signal. The claim is about local gradient alignment on those projections, not a guarantee of global optimization behavior.

What the skipped future branch costs and saves

Training and inference costs move in opposite directions depending on the backbone. For the VLM-based policy, future supervision increases training step time from 0.444 to 0.631 s (+42.1%), while inference latency stays comparable at 202.98 versus 203.77 ms per chunk relative to π0.5. For the VGM-based policy, training step time is nearly unchanged at 0.973 versus 0.975 s, while inference latency drops from 446.55 to 253.45 ms per chunk, a 1.8× speedup over FastWAM-Joint.

The asymmetry comes from what deployment retains. The VLM-based policy uses its context and action experts to denoise an action chunk, and the generation expert plus its VAE processing are unnecessary for control. The VGM-based policy processes the current observation once and caches keys and values at each transformer layer, then reuses that cache at every action-denoising step while updating only the action stream, without allocating or processing future-video tokens. Its generation expert's observation processing is kept because it supplies visual context for action generation.

So the efficiency argument is strongest for the VGM backbone, where removing future generation from the action path is what produces the speedup. For the VLM backbone the gain is robustness and supervision, not latency.

Where the action-to-future design pays off

The paper's own comparison structure points to a specific priority. If the goal is the highest reported manipulation success, FDM w/ VLM is the stronger configuration here: 99.4% average on LIBERO and 96% average on the five real-robot tasks, versus 98.1% and 83% for FDM w/ VGM. If the goal is inference speed on a video backbone, FDM w/ VGM is the relevant choice, cutting latency from 446.55 to 253.45 ms per chunk while still improving real-world average success from 78% to 83% over the architecture-matched FastWAM-Joint.

The ablation supplies the decision threshold that matters most for anyone considering this design. A future objective that only co-trains on visual prediction, without routing gradients through action features, recovers part of the gain (83% to 87% for VLM, 66% to 72% for VGM). The remaining step requires the backward connection through Ha, worth 7 and 5 points over the stop-gradient control. The paper's distinctive claim is therefore narrower than "predict the future to act better": the value sits in supervising action features with observed consequences while keeping predicted futures out of the action path.

Sources used in this answer