[CVPR 2025] Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence

Executive Summary

TL;DR: Holi-Spatial is the first fully automated pipeline capable of converting raw, unlabelled web videos into high-fidelity 3D spatially-aware datasets. By bypassing the need for human-in-the-loop annotation and specialized 3D sensors, the authors produced Holi-Spatial-4M, a dataset featuring 12,000 3DGS scenes and 1.2M spatial QA pairs. This work effectively bridges the "spatial gap" in current Vision-Language Models, showing that the key to 3D intelligence lies in the systematic fusion of geometric optimization and semantic reasoning.

Positioning: This work moves beyond incremental SOTA-chasing on static benchmarks like ScanNet. It proposes an automated annotation engine that sets a new upper bound for 3D perception and grounding, transforming video streams into a scalable source of 3D supervision.

The "Spatial Gap": Why Current Models Struggle in 3D

While current Large Multimodal Models (LMMs) excel at 2D image analysis, they often lack a true sense of 3D geometry. The root cause is the data scarcity bottleneck.

1. Lack of Scale: Manual 3D scanning is expensive and slow.
2. Domain Gap: Existing datasets like ScanNet cover only a few hundred indoor rooms with limited object categories (e.g., only 50 classes).
3. Geometric Inconsistency: Applying 2D models to video frames independently results in "flickering" or "ghosting" when those predictions are projected into 3D space.

Methodology: The Three-Stage Refinement Pipeline

Holi-Spatial solves these issues through a principled coarse-to-fine pipeline that enforces multi-view consistency.

1. Geometric Optimization (The Skeleton)

Instead of relying on noisy point clouds, the authors use 3D Gaussian Splatting (3DGS). They initialize from monocular depth priors (Depth-Anything-V3) and optimize the scene to remove "floaters"—artificially generated points that don't correspond to real surfaces. This creates a clean "surface-aligned" stage for semantic lifting.

2. Image-level Perception (The Muscles)

Using Gemini-Pro and SAM3, the system identifies objects in keyframes. To maintain consistency, they implement a dynamic class-label memory, ensuring that a "chair" identified in Frame 1 is still recognized as the same "chair" in Frame 100.

3. Scene-level Refinement (The Brain)

This is the core innovation. Initial 2D predictions are "lifted" into 3D using the rendered depths from Step 1.

• Multi-View Merge: Redundant observations of the same object are clustered using 3D IoU.
• Agent-based Verification: For ambiguous cases (confidence between 0.8 and 0.9), a VLM-based agent "zooms in" and re-evaluates the object, significantly boosting precision without sacrificing recall.

Figure 1: The Holi-Spatial pipeline—from raw video to refined 3D OBBs and captions.

Experiments: Superior Geometric and Semantic Fidelity

Holi-Spatial was tested on ScanNet, ScanNet++, and DL3DV. The results are categorical:

• 3D Detection: On ScanNet++, it achieved 81.06 AP25, compared to just 12.2 for previous SOTA LLaVA-3D.
• Depth Accuracy: The Depth F1 score reached 0.89, nearly tripling the performance of M3-Spatial (0.39).
• Spatial Reasoning: Fine-tuning Qwen3-VL on the curated QA pairs led to a 49.1% accuracy on MindCube, a massive jump from the base model's 29.4%.

Figure 2: Qualitative depth and point cloud comparison. Holi-Spatial (right) produces far cleaner geometry compared to prior baselines.

Critical Analysis & Conclusion

Takeaway

Holi-Spatial proves that high-quality 3D data can be "synthesized" from unlabelled video by using models to supervise other models (an "AI Data Flywheel"). This approach effectively democratizes 3D spatial intelligence research, as it no longer requires specialized laser scanners or thousands of human hours for labeling.

Limitations

• Computational Cost: Per-scene 3DGS optimization is still more expensive than simple feed-forward inference.
• Motion Quality: The pipeline remains sensitive to heavy motion blur or highly dynamic objects (e.g., people moving in the frame).

Future Outlook

The release of Holi-Spatial-4M provides a massive new playground for the research community. Expect to see this framework being applied to Outdoor Autonomous Driving and General-Purpose Robotics, where precise spatial grounding is a prerequisite for safe interaction.