Thinking with Spatial Code: Why 3D Perception is the Real Bottleneck for AI Video Reasoning
Thinking with Spatial Code for Physical-World Video Reasoning
The paper introduces "Thinking with Spatial Code," a framework that converts RGB video into structured, temporally coherent 3D representations (spatial codes) for physical-world reasoning. By combining a dual-encoder spatial parser with an LLM finetuned via Reinforcement Learning (RL), the model achieves new SOTA performance on VSI-Bench, surpassing proprietary models like GPT-4o and Gemini-1.5 Pro.
TL;DR
Researchers from Johns Hopkins and Stanford have challenged the "bigger is better" dogma of Multimodal Large Language Models (MLLMs). Their new framework, Thinking with Spatial Code, proves that a 4B parameter model can out-think a 230B parameter giant like Seed-1.6 on physical-world tasks. The secret? Converting video into explicit 3D "spatial codes" (coordinates, sizes, orientations) and training the LLM to process these symbols using a specialized Spatial Rubric Reward.
The Problem: AI that Sees but Doesn't Understand "Where"
Current SOTA models like GPT-4o or Gemini-1.5 are visually impressive but spatially "naive." They excel at describing what is in a frame but struggle with where things are in a metric 3D space. When asked, "If I face the table, is the washer to my front-left?", these models often hallucinate based on 2D camera views rather than reconstructing the actual 3D layout.
The authors identify a reasoning-action disconnect: models often generate the correct logic in their Chain-of-Thought but fail the final answer because they lack a grounded geometric reference.
Methodology: Parsing the World into 3D Symbols
The framework operates on a "Perception-then-Reasoning" paradigm, formalizing video understanding as an inverse graphics problem.
1. The Spatial Encoder
Instead of passing raw pixels directly to the LLM, a dedicated Spatial Encoder parses the video. It uses:
- SAM-2: For temporally consistent object tracking and segmentation.
- Depth Anything 3: To infuse 3D-aware geometric features.
- 3D Detection Head: To output a "Spatial Code" —representing label, position, size, and orientation (quaternion) for every object.
2. Thinking with Spatial Code
The LLM receives these codes as text strings. This allows the model to perform coordinate-based reasoning. For example, it can mathematically calculate unit vectors and dot products to determine relative directions, effectively acting as a geometric engine rather than an image-matching pattern recognizer.
3. Reinforcement Learning with Spatial Rubrics
To fix logic errors, the authors used GRPO (Group Relative Policy Optimization). Unlike standard RL which only rewards the correct final answer, they introduced a Spatial Rubric Reward. This rewards "perspective-based reasoning" (using the object's view) and penalizes "viewer-centric errors" (relying on the camera's perspective).
Experiments & Results: Efficiency Over Scale
The results on VSI-Bench (a hardcore spatial reasoning benchmark) were a wake-up call for the industry.
- SOTA Achievement: Thinking with Spatial Code (4B) reached 57.0% accuracy, outperforming GPT-5o (55.0%) and Gemini-2.5-Pro (53.5%).
- The 2D Boost: When provided with 2D box prompts, accuracy jumped to 60.0%, crushing the 8B version of Qwen3-VL by a significant margin.
Deep Insight: Perception Quality is the Ceiling
Perhaps the most critical finding in this paper is the Perception-Reasoning Gap. The authors tested the LLM using Ground-Truth (GT) spatial codes. With perfect 3D data, the 4B model achieved 73.2% accuracy. When using predicted codes (where perception F1 score is ~0.52), accuracy dropped to 60.0%.
The Takeaway: Scaling the "brain" (LLM) from 4B to 230B parameters yields diminishing returns if the "eyes" (Perception) are only providing blurry 2D features. True physical-world AI requires better 3D representations, not just more parameters.
Conclusion
Thinking with Spatial Code represents a shift back toward structured, symbolic representations within the era of neural networks. By forcing the model to "think" in metric 3D coordinates, it eliminates common hallucinations and provides a transparent, interpretable reasoning path for physical interactions. For future robotics and AR applications, this "spatial code" approach seems far more robust than end-to-end pixel-to-text mapping.
Limitations: The model still depends on the initial perception accuracy of the Spatial Encoder. If the encoder misses an object or miscalculates depth, the LLM—no matter how smart—will reason from false premises.