[CVPR 2026] VG3S: Elevating 3D Occupancy Prediction with Foundation Model Geometry Grounding

TL;DR

VG3S bridges the gap between high-level Vision Foundation Models (VFMs) and efficient 3D scene representation. By injecting frozen geometric priors from models like DVGT into a 3D Gaussian Splatting pipeline via a specialized hierarchical adapter, it achieves a massive 12.6% IoU boost on the nuScenes benchmark, solving the long-standing issue of fragmented object geometries in vision-only systems.

Perspective: Why Current Occupancy Models "Break"

Semantic occupancy prediction is the "holy grail" of autonomous driving perception, providing a dense volumetric understanding of the world. However, most existing models (like BEVFormer or early Gaussian-based methods) suffer from a fundamental flaw: Geometric Inconsistency.

Because their image encoders are trained on limited, task-specific datasets, they lack a "global" understanding of 3D physics. The result? Roads that look like Swiss cheese and buildings with missing corners. While VFMs (Vision Foundation Models) contain these missing priors, fine-tuning them is a computational nightmare. VG3S asks a brilliant question: Can we use a frozen VFM as a "geometric anchor" for 3D Gaussians?

Methodology: The Hierarchical Geometric Feature Adapter (HGFA)

The core innovation is the HGFA, a plug-and-play bridge that translates generic VFM tokens into high-fidelity 3D Gaussian parameters.

1. Grouped Adaptive Token Fusion (GATF)

VFMs produce layers of features with varying levels of abstraction. GATF partitions these into $K$ groups and uses an attention-based weighted summation to suppress noise while preserving the most informative geometric activations.

2. Task-Aligned Token Refinement (TATR)

Since a VFM is "task-agnostic," its features contain data irrelevant to driving. TATR uses a residual block with group-specific expansion ratios to project these features into a manifold optimized for semantic occupancy.

3. Latent Spatial Feature Pyramid (LSFP)

Occupancy requires both "the big picture" (the whole road) and "fine details" (a traffic cone). LSFP restructures the 1D tokens back into a 2D spatial grid, applying depth-wise convolutions and Squeeze-and-Excitation (SE) blocks to build a multi-resolution feature pyramid.

Experiments: SOTA Reached

VG3S was tested on the nuScenes benchmark against heavyweights like OccFormer and GaussianFormer-2.

• Quantitative Supremacy: VG3S-DVGT achieved 34.41% IoU, outperforming the previous best Gaussian-based method by a wide margin. It excels particularly in "Structural Categories" like Drivable Surfaces (42.42%) and Buildings/Manmade (17.46%).
• Generalization: Whether using DINOv2, VGGT, or DINOv3, the framework consistently improved over the baseline, proving that the HGFA adapter is truly model-agnostic.

Critical Analysis: The Power of "Frozen" Knowledge

The most impressive takeaway from VG3S is its ability to produce continuous surfaces. In the qualitative results, the baseline often misses large chunks of the road or buildings in complex intersections. VG3S, powered by the VFM's understanding of multi-view correspondence, "fills in the blanks" naturally.

Limitations: While the VFM is frozen, the Gaussian decoder and HGFA still require training. Furthermore, the performance is capped by the quality of the VFM's pre-training; a VFM that hasn't seen enough urban outdoor data might not provide the same benefits.

Conclusion

VG3S proves that the future of 3D perception isn't just about bigger models, but about smarter integration. By treating Foundation Models as reliable "geometric consultants" rather than just feature extractors, VG3S sets a new standard for efficiency and accuracy in autonomous driving scene understanding.