CityRAG: Stepping Into a Photo to Navigate the Real World

TL;DR

CityRAG is a pioneering video generative model that combines the creative power of diffusion transformers with the factual grounding of Retrieval-Augmented Generation (RAG). By querying a "memory" of geo-registered Street View data, it can transform a single photo into a long, 3D-consistent navigable video that perfectly matches the real-world geometry of the location, while preserving the specific weather and dynamic conditions of the input image.

Background: Memory vs. Hallucination

In the current landscape of AI, models like Sora or Wan 2.1 can dream up spectacular videos. However, if you ask them to "drive down 5th Avenue in a snowstorm," they will hallucinate a generic city that looks like New York but doesn't match New York. For autonomous driving or virtual tourism, "close enough" isn't enough—we need geographical precision.

The Core Insight: Semantic Disentanglement

The fundamental challenge addressed by the Google and Stanford researchers is disentanglement. A real-world scene consists of:

1. Static Elements: Buildings, roads, and permanent infrastructure.
2. Transient Elements: Lighting, weather, pedestrians, and moving vehicles.

CityRAG solves this by using temporally unaligned training data. By providing the model with two views of the same street taken months apart (e.g., a sunny morning vs. a rainy night), the model is forced to learn that the buildings stay the same while the "style" and "objects" change.

Methodology: The RAG-Enhanced DiT

The architecture builds upon the Wan 2.1 (14B) Diffusion Transformer. The researchers added two critical conditioning paths:

• Trajectory Conditioning: Camera poses (extrinsics) are injected into the transformer blocks via a zero-initialized projection layer, allowing precise control over the "virtual walk."
• Geospatial Conditioning (The RAG Part): The model retrieves Street View frames near the target trajectory. These are processed through a dedicated cross-attention branch, serving as a "structural anchor."

Figure 1: The CityRAG Pipeline. Note the dual-path approach separating the first image (Style/Dynamics) from the retrieved geospatial memory (Structure).

Robustness in the Wild

One of the most impressive feats of CityRAG is its ability to handle stitched trajectories. If a user wants to turn right at an intersection, but the retrieved Street View data only goes straight, CityRAG can "stitch" together a second video from the cross-street. Even with this discontinuous data, the model perceives the underlying 3D layout and generates a smooth, consistent turn.

Figure 2: Qualitative Comparison. CityRAG (Ours) maintains the black car from the first frame and accurately renders buildings that only appear much later in the reference "memory" frames.

Experiments & Results

The quantitative gap between CityRAG and previous methods is stark. Using metrics from the Novel View Synthesis (NVS) community (PSNR, SSIM, LPIPS), CityRAG outperforms baselines across the board.

The FID improvement (from 61 to 16) indicates a massive leap in visual realism and distribution alignment. More importantly, user studies confirmed that CityRAG is the only model currently capable of producing videos that are both smooth continuations of the starting frame and faithful to the physical location's layout.

Critical Analysis & Conclusion

While CityRAG is a landmark step toward "World Simulators," it has limitations. The training data (Street View) is inherently biased toward driving paths and clear weather. The model does not yet natively respond to text prompts for fine-grained editing (e.g., "add a parade to this street"), as it prioritizes visual and spatial grounding.

Takeaway: CityRAG proves that generative models can be "tethered" to reality. By treating the world's existing imagery as a retrievable database, we can move past pure AI hallucination toward reliable, grounded simulations for the next generation of robotics and AI agents.