TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens
TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens
TokenGS is a feed-forward 3D Gaussian Splatting (3DGS) framework that introduces an encoder-decoder architecture with learnable tokens to decouple primitive prediction from input resolution. By directly regressing 3D coordinates instead of depth along rays, it achieves state-of-the-art (SOTA) performance in sparse-view reconstruction and handles dynamic scenes with temporal consistency.
TL;DR
TokenGS revolutionizes feed-forward 3D Gaussian Splatting (3DGS) by moving away from pixel-aligned depth prediction. Instead, it uses an encoder-decoder Transformer with learnable tokens to directly regress 3D coordinates. This shift allows the model to handle fewer, more efficient Gaussians, remain robust to noisy camera poses, and naturally support dynamic scene "bullet-time" reconstruction through token-based temporal conditioning.
Background: The Pixel-Alignment Trap
In the current SOTA for feed-forward 3D reconstruction (like GS-LRM or MVSplat), the number of 3D Gaussians is usually a function of the input resolution—often one Gaussian per pixel or patch. While intuitive, this "pixel-aligned" approach has three fatal flaws:
- Redundancy: 32 images at 512p resolution result in >8 million Gaussians, regardless of whether the scene is a complex forest or a blank white wall.
- Rigidity: Predicting depth along rays means if your camera pose is slightly off, your 3D geometry "shatters" because the rays don't intersect correctly.
- Visibility: You can't easily predict what you don't see (occlusions) because there are no "rays" pointing there.
Methodology: The Power of Tokens
TokenGS introduces a paradigm shift by treating Gaussians as learnable entities (tokens) rather than pixel properties.
1. Direct 3D Coordinate Regression
Instead of saying "this pixel has a depth of 5.0", TokenGS says "this token represents a Gaussian at (x, y, z)". This is supervised purely via rendering loss. To prevent Gaussians from "wandering off" into empty space (the zero-gradient problem), the authors introduce a Visibility Loss (Eq. 2) that penalizes particles that don't project onto any supervision view.
2. Encoder-Decoder Architecture
The model uses a ViT encoder to extract image features and a DETR-like decoder where Gaussian tokens cross-attend to those features. This decouples the output complexity from the input resolution.
Figure 1: The TokenGS Pipeline. Notice how the Gaussian tokens act as queries to extract geometry from multi-view image features.
3. Static-Dynamic Decomposition
By splitting tokens into "Static" and "Dynamic" sets and using causal attention masking, the model can separate a scene's background from moving objects. Dynamic tokens receive time-embeddings, enabling smooth "bullet-time" interpolation.
Experiments & Results
TokenGS demonstrates superior "regularization." Because the Gaussians aren't forced onto specific rays, the reconstructed surfaces are smoother and free of the "spiky" artifacts prevalent in depth-based models.
Robustness and Efficiency
- Redundancy Reduction: TokenGS achieves higher PSNR with 72% fewer Gaussians than baselines on the DL3DV dataset.
- Noise Tolerance: When 10 degrees of rotation noise is added to camera poses, TokenGS maintains its performance far better than GS-LRM, which relies on precise ray geometry.
- Test-Time Scaling (TTS): A unique feature of TokenGS is "Token-Tuning." You can fine-tune the token embeddings (not the whole network) at test time. This improves quality without destroying the model's learned priors.
Table 1: Performance on RealEstate10K. TokenGS (4096 tokens + TT) sets a new SOTA with 28.82 PSNR.
Critical Insight: Why it Works
The "magic" lies in Slot Specialization. In pixel-aligned models, every pixel is forced to contribute a Gaussian. In TokenGS, specific tokens learn to "specialize" in certain types of geometry or spatial regions. As shown in the qualitative results, this leads to a more balanced distribution of primitives—more Gaussians are sent to high-detail areas, and fewer to flat surfaces.
Figure 2: Qualitative comparison showing the elimination of spiky artifacts (GS-LRM) through TokenGS's decoupled prediction.
Conclusion & Future Work
TokenGS is a significant step toward "foundation models" for 3D reconstruction. By breaking the link between pixels and primitives, it offers a flexible, robust, and computationally efficient framework.
Limitations: The primary bottleneck remains the cost of Token-Tuning at inference time and the inherent difficulty of scaling to city-scale environments. However, the ability to recover emergent attributes like scene flow suggests that TokenGS is not just a renderer, but a true geometry learner.