AnyRecon: Scaling 3D Reconstruction to Arbitrary Views via Video Diffusion
AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model
AnyRecon is a scalable 3D reconstruction framework that leverages a video diffusion model to synthesize novel views from arbitrary, unordered sparse inputs. It achieves state-of-the-art performance in both view interpolation and extrapolation by combining explicit geometric control with a flexible conditioning mechanism.
TL;DR
AnyRecon is a breakthrough framework designed to turn sparse, "casual" photo captures (like those from a smartphone) into consistent 3D scenes. By re-engineering video diffusion models to handle unordered inputs and integrating a recursive 3D geometry memory, it achieves high-fidelity reconstruction with a 20x speedup compared to traditional iterative refinement methods.
Background: The Sparse-View Bottleneck
Traditional 3D reconstruction (like NeRF or Gaussian Splatting) requires dense image coverage to work. When views are sparse, the geometry becomes "ill-posed"—there are too many ways to fill the gaps. Recent "generative" approaches try to solve this by using Diffusion Models to "hallucinate" the missing views. However, existing models suffer from a "Conditioning Gap": they usually only look at the first and last frames, losing the context of everything in between, or they assume the camera moves in a smooth, sequential line.
AnyRecon addresses this by asking: What if we treat all captured views as a global memory that any new viewpoint can query?
Methodology: The Core Innovations
AnyRecon shifts from a standard "video player" logic to a "3D memory" logic.
1. Global Scene Memory & Non-Compressive Encoding
Prior models use 3D-VAEs that compress video over time. This works for movies but fails for 3D reconstruction because it "smears" features between different viewpoints. AnyRecon removes this temporal compression, using a pure 2D VAE to ensure every pixel in the latent space maps exactly to a specific coordinate in 3D space. It then prepends all captured "reference" images to the transformer sequence, allowing the model to attend to any relevant visual detail globally.
2. Geometry-Aware Conditioning
The secret to AnyRecon’s stability is a "closed loop":
- Reconstruction to Generation: It renders a coarse point cloud from existing views to give the diffusion model a geometric "sketch."
- Visibility-Driven Retrieval: Instead of picking the "closest" image, it analyzes the point cloud to see which captured images actually see the surfaces visible in the target view. This avoids being distracted by occluded images.
- Generation to Reconstruction: Once new views are generated, they are back-projected to update and "fill in" the 3D point cloud, which then guides the next segment of the trajectory.
Efficient Synthesis: 4-Step Distillation
Generating 40 frames of high-resolution video is computationally heavy. AnyRecon employs Distribution Matching Distillation (DMD2). It compresses the standard 50-step denoising process into just 4 steps. When combined with Sparse Attention (limiting the search area of each pixel), the system generates full scenes in under two minutes.
Experiments and Performance
The model was tested on challenging datasets like DL3DV and Tanks and Temples.
- Interpolation: Filling gaps between distant frames.
- Extrapolation: Hallucinating the scene as the camera moves into unobserved territory.
Key Results:
- Accuracy: Outperformed baselines like ViewCrafter and Uni3C by nearly 5dB in PSNR.
- Visual Quality: Significantly reduced "floaters" and color shifts common in sparse-view 3DGS.
- Speed: Completed 40-frame sequences in ~105 seconds, whereas refinement-based methods (like Difix3D+) took up to 20 minutes.
Critical Analysis & Takeaways
The brilliance of AnyRecon lies in its structural honesty. By removing temporal compression and enforcing visibility-aware retrieval, it respects the laws of 3D geometry rather than just treating 3D as a "consistent video" problem.
Limitations: It still relies on a "base" geometric coherence. If the initial sparse views have zero overlap, the first point cloud will be garbage, and the diffusion model won't have the "anchor" it needs to start the loop.
Future Outlook: AnyRecon paves the way for "Reconstruction-on-the-fly," where a user can record a few snippets of a room and have a high-fidelity 3D asset generated in near real-time, capable of handling complex office or outdoor environments.