Why pixel motion is the wrong currency for manipulation
A world action model is asked to answer two linked questions at once: given an instruction, what motion should the robot execute, and how will that motion change the surrounding world? Most existing models answer both through images or visual latents, typically by building on pretrained video generators. The paper argues that this choice imports a mismatch. Robotic interaction happens in metric three-dimensional space, while an image is a view-dependent projection of that space.
The consequence is that the same robot motion, object displacement, or contact event can produce different pixel changes across views, and equal pixel displacements need not correspond to equal physical distances. Camera intrinsics and camera-to-robot poses vary across embodiments, so the image-to-motion mapping shifts with the setup. The paper also cites reported sensitivity of video-based world action models to viewpoint and robot-state changes, and notes that a photorealistic prediction may still misrepresent object displacement, deform rigid geometry, or let static regions drift. In that framing, metric motion and contact relationships stay implicit, which is exactly what spatial reasoning, prediction, and control need explicitly.
One metric frame for hands and grippers
UNITAS instead fixes a single metric frame within each prediction window. The context at time t contains the instruction, multi-view observations, camera calibration, and historical action flow; the calibration maps camera coordinates into a world frame that stays fixed for that window. Robot demonstrations use a robot-base or shared embodiment frame, while egocentric human demonstrations use the camera frame of the window anchor, held fixed throughout the window.
Within that frame the model defines two complementary quantities. Action flow represents human hands and robot grippers as 3D point trajectories, sampled as displacement from each query point's current position over future timestamps. Scene flow describes scene-point displacements conditioned on those trajectories. Because both are expressed as metric displacements in the same frame, the representation is shared across robot embodiments and human hands rather than being tied to one camera view. A separate action branch predicts an executable end-effector action chunk, so the joint distribution factorizes into action flow, the command chunk, and scene flow conditioned on action flow.
Grounding visual tokens in world coordinates
For the 3D frame to matter, the visual tokens must know where they are in it. Each visual token receives a world-aligned 3D position embedding. When depth is available, the model back-projects each pixel using the depth along the optical axis and the camera intrinsics, transforms the result into world coordinates with the camera-to-world transform, encodes the position with a fixed Fourier map, and adds it to the visual token.
Depth is not always available, so the paper adds a fallback. Without observed depth, the embedding is computed from a weighted combination of positional features along the camera ray: for each patch, 64 depths are drawn uniformly from 0.1 to 2.0 m, back-projected, and scored against the patch feature, with keys encoding camera-frame positions and values encoding the same candidates in the world frame. Where depth is reliable, the routing weights are supervised to match Fourier features of the true world positions. This lets UNITAS pretrain jointly on RGB and RGB-D data and run without depth at inference, which is the practical condition under which many real deployments operate.
One token per trajectory keeps time physical
A trajectory is a long object, and naively tokenizing every point at every timestep would make the sequence unwieldy. The paper's physical-time trajectory tokenizer encodes each point trajectory as a single token anchored at its current 3D position, with elapsed time expressed as a physical quantity rather than an index into a frame stack. The tokenizer is first trained to reconstruct historical and future action flow and available scene trajectories, then frozen.
The two experts then learn through rectified flow. For a clean target, either a normalized trajectory latent or an action chunk, noise is mixed in at level gamma and the network regresses the difference; action flow and the action chunk share gamma but use independent noise, while scene flow draws its own gamma. During training, scene prediction conditions on clean demonstrated action-flow latents. The attention mask enforces the intended information flow: scene tokens attend to context, clean action flow, and one another, but not to noisy action flow, so scene prediction conditions on action flow without leaking into action-flow denoising. Action chunk tokens attend only to observation keys and values, the current end-effector state, and one another, which is what allows policy-only inference to skip both future-flow branches entirely.
Scene prediction gains are measured in centimeters
The scene-prediction claim rests on RoboTwin, where the paper reports the best action-conditioned scene prediction among compared methods, with up to 49% lower displacement errors than PointWorld. The evaluation is deliberately built to be comparable across model families. Each episode contributes a fixed window near its midpoint containing the current frame and 20 future recorded frames, and initial scene points are sampled on a 1.5 cm voxel grid. All methods share initial queries, point identities, ground-truth trajectories, and robot-motion conditions.
Video-based baselines are handled by projecting the shared queries into the initial image using camera calibration, tracking only in-frame projections, and transforming tracked trajectories into world coordinates anchored at the shared initial queries. Depth scale for Ctrl-World and inverse-depth affine calibration for X-WAM are estimated from initial observed depth and fixed throughout the video, and no future ground-truth depth or trajectories are used for alignment. Scoring separates moving from static points with a 1 cm threshold on maximum ground-truth displacement over the 20 future frames, then reports average displacement error over valid point-frame pairs, final displacement error for moving points at frame 20, and the fraction of moving-point-frame pairs within 1 cm. These are metric errors in centimeters, not pixel differences, which is the point of the comparison.
Success rates and what they do not settle
On manipulation success, UNITAS reports 99.8% average on LIBERO, including 99.4% on LIBERO Long, and an average of 85% across real-world tasks. The figure reported in the paper's overview also lists 92.9% on RoboTwin Clean and 93.0% on RoboTwin Randomized, 56.2% success rate and 67.6% progress score on VLABench, and 82.1% on the LIBERO-Plus camera split, 94.2% on the robot split, and 89.1% overall. These numbers answer a different question from the displacement errors: they measure whether the executed action chunk completes the task, not whether predicted scene points land within a centimeter.
The paper's own closed-loop consistency test links the two. On three RoboTwin tasks under Clean and Randomized conditions, six settings with 10 trials each, the point dynamics expert generates an action flow while the action expert separately generates an executable command chunk from the same observation and instruction; the dynamics expert then predicts scene flow conditioned on its generated action flow, and those predictions are compared with the ground-truth scene trajectories produced by executing the action expert's command chunk in the simulator. That design measures internal action-scene agreement, not agreement with a human teleoperator's trajectory, and the paper notes that all episodes are retained regardless of task outcome while rounds without valid moving points are excluded only from moving-point metrics.
Where the 3D claim stops paying off
The strongest evidence for the 3D-native formulation is the combination of centimeter-level scene-prediction errors on RoboTwin and high manipulation success in both simulation and real tasks, since a purely visual representation would have to recover metric displacement implicitly. The paper's discussion frames this as making interaction geometry an explicit prediction target, connecting world modeling with action learning and giving a shared representation across embodiments and views. That is the authors' interpretation of why the numbers look as they do; the packet does not include an ablation isolating how much of the success comes from the metric frame versus the policy branch alone.
The scope conditions matter for anyone deciding whether to build on this. The real-world average of 85% is reported across tasks without a per-task breakdown in the packet, and the LIBERO-Plus camera split at 82.1% is visibly below the robot split at 94.2%, which is the setting where viewpoint variation is the intended stressor. The paper also states that extending the foundation to reliable prediction over longer interactions and broader real-world deployment remains a direction rather than a demonstrated result. The practical reading is therefore specific: when the decision hinges on whether predicted scene motion is physically correct in centimeters, the RoboTwin displacement comparison against PointWorld is the relevant evidence; when it hinges on task completion under a fixed embodiment, the LIBERO and real-world success rates are; and when the camera viewpoint itself is expected to shift, the LIBERO-Plus camera split is the number to weigh first.
Sources used in this answer
UNITAS: A 3D-Native World Action Model for Embodied Manipulation
UNITAS represents observations, actions, and scene dynamics in a shared metric 3D frame, reporting the best action-conditioned scene prediction on RoboTwin among compared methods with up to 49% lower displacement errors than PointWorld, 99.8% average success on LIBERO, and 85% average across real-world tasks with 1.7B parameters.