Synthetic data nearly matches real demonstrations
The paper's central claim is that a real-world manipulation policy can be trained without a single human teleoperation demonstration. Grounded-Dreamer builds synthetic trajectories from a handful of real images, and the resulting policies are evaluated on seven real-world tasks: pressing a button, closing a drawer, pick and place, sweeping trash, slicing a sausage, putting an object in a drawer, and one further pick task.
The most decision-relevant comparison is against real demonstrations at the same budget. With 50 trajectories per task, the π0.5 policy trained only on Grounded-Dreamer data reached 55%, 75%, 55%, 50%, 60%, 60% and 30% success across those tasks, while the real-only baseline reached 65%, 75%, 65%, 60%, 60%, 75% and 45%. The synthetic-only policy therefore lands close to, and on pressing a button exactly at, the real-data baseline, but it does not beat it outright. The Diffusion Policy backbone shows the same pattern at lower absolute levels: 45%, 50%, 45%, 35%, 45%, 55% and 25% for synthetic-only versus 50%, 60%, 60%, 50%, 50%, 70% and 35% for real-only.
That gap matters for how the result should be read. The paper demonstrates that teleoperation can be removed from the data pipeline at a modest cost in success rate, not that synthetic data is a strict substitute for real demonstrations at equal volume.
Two failure modes the pipeline is built to avoid
The paper frames the data problem as a trade-off. Real teleoperation gives high-fidelity, action-labeled demonstrations but is limited by human labor. Simulation gives cheap action-labeled trajectories but suffers a visual sim-to-real gap, and the paper's simulation-only baseline confirms how severe that is: with 50 trajectories per task, the Diffusion Policy trained in simulation alone scored 10%, 10%, 0%, 5%, 5%, 0% and 0% on the real tasks.
Generative video world models offer the missing visual realism, but pretrained models are trained largely on human-centric video and can produce what the paper calls physical hallucinations. The authors separate two kinds: morphological hallucinations, such as invalid robot kinematics or infeasible joint configurations, and spatial-depth ambiguities, where an apparent 2D contact does not correspond to a valid 3D interaction. A generated video can therefore look convincing while depicting a grasp that could not physically happen. Grounded-Dreamer's design is essentially an answer to that specific failure, not to data scarcity in general.
Real images are rebuilt as a physics-checkable scene
The pipeline begins with a small set of real reference images: typically one scene overview and one to four object perspectives. From these, the system reconstructs a simulation scene containing the robotic arm and workspace table, calibrates the simulated camera to match the real camera's field of view, and rebuilds task-relevant objects. Rigid objects are reconstructed in 3D and rescaled to real-world dimensions; articulated objects are decomposed into components and reassembled into a kinematic structure inside the simulator using predefined connectivity rules and joint specifications.
This reconstruction is what makes later verification possible. Because the generated data can be replayed in a scene that is geometrically aligned with the real setup, a trajectory can be tested for physical validity rather than judged only by how realistic its video looks. The paper is explicit that pixel-level fidelity in task-irrelevant regions is unnecessary, and it reuses off-the-shelf assets for tabletop textures and backgrounds.
Masking task-irrelevant pixels is close to essential
To post-train the video world model, the authors first collect 100 to 150 simulation trajectories per task using an LLM-based agent that writes and executes data collection scripts. They then apply interaction-aware masking, which keeps only the robotic arm and the objects being manipulated in the simulation frames. The stated purpose is to push the world model and the inverse dynamics model toward modeling interaction dynamics rather than background appearance.
The ablation shows this is not a minor detail. Removing masking collapses performance: the Diffusion Policy trained on unmasked synthetic data scored 0%, 5%, 0%, 0%, 0%, 0% and 0% across the seven tasks, against 45%, 50%, 45%, 35%, 45%, 55% and 25% with masking. The π0.5 backbone shows the same collapse, from 55%, 75%, 55%, 50%, 60%, 60% and 30% down to 5%, 0%, 10%, 0%, 10%, 5% and 0%. The paper attributes this to distorted pixels and unrealistic scene deformations in the generated videos when raw simulation frames are used for fine-tuning.
Replaying trajectories in simulation removes false successes
After the fine-tuned world model generates rollout videos from augmented initial frames, an inverse dynamics model converts those videos into action trajectories. The trajectories are then replayed in the reconstructed simulator and labeled optimal or sub-optimal according to task success criteria. The paper's illustrative case is a gripper placed incorrectly: the physics simulation registers a failure, while the world model had generated a hallucinated successful outcome.
The ablation quantifies how much this filtering matters. Training on 100% verified trajectories gives the full results reported above; at 80% verified, π0.5 success falls to 35%, 60%, 50%, 35%, 45%, 50% and 20%; at 50% verified it falls further to 25%, 35%, 25%, 20%, 25%, 30% and 10%. Success rises monotonically with the share of verified data, and the paper describes a precipitous drop at 50% sub-optimal data. Verification is therefore the component that converts a visually plausible generator into a usable data source.
Synthetic variety helps most when the environment shifts
The strongest result for synthetic data is not in-distribution accuracy but robustness. Before generating trajectories, the pipeline randomizes object poses and uses an image-editing model to composite simulated objects onto real backgrounds, then augments lighting and tablecloth textures. This produces initial frames that vary in ways real collection would require repeated physical re-collection to match.
Under out-of-distribution conditions, the real-only baselines degrade sharply: average success fell to 7.1% for Diffusion Policy and 16.4% for π0.5. Policies trained on verified world-model data held up much better, reaching 40%, 50%, 35%, 30%, 40%, 45% and 15% for Diffusion Policy and 45%, 60%, 55%, 40%, 60%, 50% and 30% for π0.5. The paper's interpretation is that the real-only models were constrained by limited training-set diversity, while the synthetic pipeline supplies environmental variety autonomously. This is the scenario where the approach's advantage is clearest.
A few real trajectories plus synthetic data beat either alone
The co-training regime is the paper's most favorable configuration. Combining a minimal set of real demonstrations with 50 Grounded-Dreamer trajectories outperformed both the synthetic-only model and matched or exceeded the 50-sample real-only baseline. For π0.5, co-training reached 70%, 80%, 65%, 55%, 75%, 70% and 45%, above the real-only 65%, 75%, 65%, 60%, 60%, 75% and 45% on most tasks. For Diffusion Policy it reached 55%, 65%, 50%, 45%, 55%, 60% and 40%, also above real-only on most tasks.
The contrast with low-data real training is stark: real-only models trained on the minimal set averaged 8.6% for Diffusion Policy and 14.3% for π0.5. The authors describe the few real trajectories as domain anchors that align the synthetic distribution with the real domain. Read against the tables, the practical priority is not choosing synthetic over real data but using a small real set to anchor a much larger verified synthetic set.
Returns flatten near 150 trajectories per task
Scaling synthetic trajectories improves success on the two tasks examined, closing a drawer and putting an object in a drawer, but performance begins to plateau at roughly 150 trajectories per task, with further additions yielding only marginal gains. The paper notes that its experiments used hundreds of trajectories per task and that scale was constrained by available compute rather than by the algorithm, so the plateau is an observed result at this scale, not a proven ceiling.
Two further limits bound the claim. The pipeline still requires a minimal one-time collection of reference images, so it removes teleoperation but not all real-world input. And the image-editing model that synthesizes initial frames succeeds about 65% of the time, which the authors treat as a baseline expected to improve as generative models mature. The paper also cautions that biases in world models and the safety of policies trained on synthetic data need attention before deployment in human-centric environments.
Taken together, the evidence supports a specific priority rather than a blanket replacement claim: use verified synthetic data to supply diversity and volume, keep a small real set as a domain anchor, and treat replay verification and interaction-aware masking as the components that make the data usable at all.
Sources used in this answer
Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
Policies trained only on verified synthetic trajectories from a video world model reached real-world success rates close to, but generally below, policies trained on 50 real demonstrations per task, and a small real set combined with synthetic data outperformed either alone.