AnyCamVLA: Solving the Viewpoint Brittleness of Robot Foundation Models via Zero-Shot Adaptation

TL;DR

Robot foundation models (VLAs) are incredibly smart until you move the camera 3 centimeters. AnyCamVLA fixes this "viewpoint brittleness" without retraining the model. By using a real-time Large View Synthesis Model (LVSM), it virtually warps any camera feed (even from an iPhone) back to the original training viewpoint, allowing a frozen robot policy to function perfectly in unfamiliar hardware setups.

Background: The "3cm Failure" Problem

Generalist Vision-Language-Action (VLA) models like OpenVLA or ${\pi }_0$ have revolutionized robotics by bringing Internet-scale semantic understanding to the factory floor. However, they share a fatal flaw: Overfitting to the training viewpoint.

In real-world deployment (homes, offices), cameras are rarely mounted in the exact same spot twice. Prior research shows that a minor shift in a wrist-mounted camera can cause a policy’s success rate to plummet from 90% to under 30%. The industry standard fix—data augmentation—is a "brute force" method that requires massive data and often leads to catastrophic forgetting, where the model loses its original skills while trying to learn new perspectives.

The AnyCamVLA Insight: Virtual Camera Alignment

Instead of teaching the robot's "brain" to be invariant to viewpoints, why not just change the "eyes"?

AnyCamVLA introduces a plug-and-play adapter that sits between the camera and the VLA. At every control step, it performs two main tasks:

1. Input Sensing: It takes the "wrong" view from the test-time camera.
2. Neural Warping: Using a feed-forward LVSM, it synthesizes what the scene would look like from the training camera's perspective.

Method Architecture

The beauty of this approach lies in its simplicity. By keeping the VLA policy entirely frozen, the system preserves all the pre-trained semantic reasoning acquired from billions of internet images.

Figure 1: The framework runs at 30Hz, synthesizing training-view images for a frozen 10Hz VLA policy.

Experimental Evidence: Outperforming the Baselines

The authors tested AnyCamVLA against both fine-tuned models and geometry-aware models (like GeoAwareVLA) on the LIBERO benchmark.

1. Superior Robustness

While standard policies (OpenVLA, ${\pi }_{0.5}$) failed under "Large" perturbations, AnyCamVLA maintained a nearly flat performance curve.

Table 1: Success rates across LIBERO suites. Note the consistent 90%+ performance of "Ours-π" even under Large (L) camera shifts.

2. Efficiency vs. Fine-tuning

One of the most striking findings is the failure of task-specific fine-tuning. As shown in the paper's ablation, fine-tuning a model on a new viewpoint for "Task A" actually decreased its performance on "Task B." AnyCamVLA avoids this entirely because it doesn't touch the policy's weights.

3. Real-World Versatility

The researchers didn't stop at simulations. They successfully deployed AnyCamVLA using:

• Varying Extrinsics: Moving the fixed camera stand.
• Varying Intrinsics: Changing camera types (ZED vs. RealSense).
• Dynamic Views: A human holding an iPhone and moving it around while the robot was working.

Discussion: The Future of Modular Robotics

AnyCamVLA represents a shift toward modular intelligence. Instead of building one massive end-to-end model that must learn everything (geometry, semantics, and control), we can use specialized modules:

• LVSM for Geometric Invariance (Low parameter count, high speed).
• VLA for Semantic Reasoning and Control (High parameter count, frozen).

Limitations: The model currently struggles when the target viewpoint has large occluded regions. If the camera can't "see" behind an object, the synthesis model must hallucinate pixels, which can lead to inaccuracies. Furthermore, there is a ~30ms latency tradeoff—though this is well within the limits of current 10-20Hz robot control loops.

Conclusion

AnyCamVLA proves that we don't need a "perfect" robot policy that understands all of 3D space. We just need a good "translator" that can map the messy real world back into the narrow visual domain the robot was trained to understand. For any developer looking to deploy VLAs in unstructured environments, this zero-shot adaptation is the most efficient path forward.

Core Takeaway: Stop fine-tuning your VLAs for every new camera angle. Use view synthesis to adapt the world to your model, not your model to the world.