The Invisible Hand of Physics: Do Video Models Know More Than They Show?

TL;DR

Recent headline-grabbing video models like Sora and Veo are often called "world simulators," but do they actually understand physics? This paper reveals a fascinating paradox: even when a video model generates physically impossible movement, its internal "brain" (the Transformer blocks) often contains a highly accurate signal of physical reality. By reversing the diffusion process, researchers found that these models encode physical laws as a byproduct of learning to denoise, reaching up to 81% accuracy in identifying physical errors—smashing the performance of models specifically trained to understand video representations.

Problem: Visual Realism vs. Physical Logic

The AI community is currently split: are Scaling Laws leading us to a true simulation of the world, or are we just building incredibly sophisticated "stochastic parrots" for pixels?

Prior work shows that while diffusion models can generate a ball falling, they often fail to extrapolate constant acceleration or momentum conservation if the scenario deviates from their training data. Furthermore, diffusion models use VAEs (Variational Autoencoders) that are trained purely to reconstruct images, not to understand the "soul" of the scene. If the input (VAE) doesn't have physics, and the output (generated video) often breaks physics, where—if anywhere—does the knowledge of the physical world reside?

Methodology: Mining the Denoising Trajectory

To peek under the hood, the authors developed a technique to map real-world videos (with known physical labels) back into the "thoughts" of the model.

1. Reverse Sampling (The Inquiry): They took real videos of balls falling or colliding and used the model's own math in reverse to find the exact "noise" that would have generated that video. This creates a latent trajectory.
2. Linear Probing (The Detective): At every step of this journey (from clear video to noise), they extracted the hidden states of the Transformer and trained a simple linear layer (a probe) to answer: "Is this video physically plausible?"
3. Causal Intervention (The Trial): To prove these layers actually use this info, they injected noise into specific blocks during generation to see which ones caused the model to "lose its grip" on physical logic.

Figure 1: The framework for reverse sampling and probing internal activations.

The "Emergence Zone": Physics as a Byproduct

The findings were stunning.

• Higher Accuracy than Specialists: Video Diffusion Models (like WAN and CogVideoX) proved better at physical reasoning than V-JEPA, Meta’s high-profile model specifically designed for video understanding.
• The VAE is Blind: Probing the VAE latents directly yielded nothing better than a coin flip (50% accuracy). This proves the "physics" isn't in the input; it is constructed by the diffusion transformer as it learns to move through the latent space.
• The Middle Child is the Smartest: Physical information is not found at the very beginning or end of the model. It peaks in the intermediate layers (blocks 15-25).

Figure 2: Diffusion models (WAN, LTX, CogVideoX) vs. representation baselines (V-JEPA, VideoMAE).

Deep Insight: Beyond Binary Logic

The model doesn't just know "right vs. wrong." When tested on parabolic motion, the internal states could encode the exact initial velocity ($v_0$) and position ($x_0$) of objects with near-perfect precision ($R^2\approx 0.99$).

Interestingly, there is a Compression-Intelligence Trade-off. Smaller, more "compressed" models like WAN-1.3B actually showed clearer physical signals than larger models. The theory? Larger models have so much room they "waste" dimensions on high-frequency textures (like the shimmer of water), while smaller models are forced to prioritize the core "skeleton" of the scene: the physics.

Figure 3: Regression results for initial position and velocity shows deep physical encoding.

Critical Analysis & Conclusion

This research changes the narrative on "World Simulators." It suggests that the difficulty in generating physical videos isn't a lack of knowledge inside the model, but perhaps a failure in the sampling or decoding process.

Takeaways:

• Internal Knowledge > External Output: Models "know" more about the world's constraints than they are currently able to express in their generated pixels.
• The Flow is Key: Physics is a property of the trajectory (the path from noise to data), not just the final image.
• Future Impact: These internal "physics signals" could be used as a "guidance" mechanism—filtering out unphysical generations before the user even sees them.

Limitations: The authors acknowledge that while the signal is "linearly decodable," it doesn't mean the model is "reasoning" with F=ma in a human sense. It is a statistical representation of the "Flow" of the universe, captured in the weights of a billion-parameter Transformer.