Can VLA policies adapt to arbitrary camera counts and unseen poses?

Fixed camera setups make VLA policies brittle

Vision-Language-Action models turn visual observations, a language instruction, and proprioceptive state into robot actions. In conventional single- or multi-view VLAs, the visual input is a fixed tuple of images: the number of views and each camera's pose are baked into the policy input during training. The paper argues that adding a camera or moving one changes the visual token distribution, so the policy degrades exactly when a new platform, workspace, or task forces a different camera layout.

The paper's motivating figure makes the stakes concrete: the same task can succeed at 48% from one view and 51% from an alternative view, or 67% versus 60% in another comparison, depending on which view is used. Those numbers are illustrative of view sensitivity rather than a controlled benchmark, but they frame the design goal: a single policy should stay valid as the camera configuration varies at deployment.

Scene tokens decouple cameras from action learning

VersaCamVLA splits the problem into two stages. In Stage I, a scene encoder takes any non-empty set of posed RGB views and produces a fixed number of learnable scene tokens. Each view is tokenized as RGB patches concatenated with Plücker ray embeddings, which encode camera geometry; the tokens from all views are then processed jointly with the scene tokens through a Transformer encoder. Because the scene tokens are a fixed set that attends to a variable-length source sequence, their size does not depend on how many cameras are available.

Supervision comes from target-view prediction. A decoder, used only during Stage I, queries the scene tokens with target-camera rays and reconstructs RGB, semantic, and edge signals at those target views. Semantic targets come from a simulator segmentation sensor in simulation and from a SAM3 segmenter fine-tuned on the real-robot data; edges come from a Canny detector. This forces the tokens to carry view-consistent spatial information that supports prediction at poses the encoder did not see.

In Stage II the decoder is discarded, the scene encoder is frozen, and a lightweight spatial encoder compresses the 3×32×32 grid of 3072 scene tokens into a 3×8×8 representation with 192 tokens. Those compact tokens enter a pretrained π0.5 VLA as a supplementary visual condition, while the base policy keeps its standard input-output form. No explicit 3D reconstruction or novel-view rendering is required at deployment.

Wrist-camera motion supplies free pose diversity

Learning a camera-configurable interface needs training data that covers diverse camera placements, and static tabletop cameras are sparse in that respect. Wrist-Augmented Pose Sampling (WAPS) exploits the fact that wrist cameras sweep a much richer pose distribution as the robot moves. For each training frame, the source set is drawn independently from the two camera types: a random number of static views between 1 and the total available, and each wrist view included with probability 0.5. The target set is always the full camera pool, so the decoder is trained to render every available view regardless of what the encoder received.

This yields two complementary signals on the same scene tokens: self-reconstruction on source views preserves visual detail through the latent bottleneck, while cross-view prediction on held-out views forces the tokens to support poses absent from the encoder input. On RoboTwin 2.0, each scene provides eight available views, and WAPS resamples the source subset at every frame.

Multi-signal supervision beats RGB-only targets

The per-task ablation on RoboTwin 2.0 DR isolates the supervision choice. With RGB-only targets, the average success rate across the listed tasks is 20.94%; with the full multi-signal objective combining RGB, semantic, and edge targets, the average rises to 29.50%. Individual tasks move unevenly — Open Microwave goes from 12% to 54%, Stack Blocks Three from 26% to 35%, while Blocks Ranking Size drops from 6% to 9% in the other direction on a single task — so the aggregate gain is the more reliable reading than any one row.

A second ablation compares the spatial encoder's compression style. Pooling averages 50.13% on RoboTwin 2.0 Clean, while convolutional compression averages 55.63%. The paper's design uses convolutional operations for channel projection, local aggregation, spatial downsampling, and output projection, which is consistent with that comparison. These two ablations answer different questions — what the scene tokens are supervised to contain, versus how they are compressed for the policy — and should not be merged into a single claim about the method.

Robustness holds across counts and unseen poses

The headline claim is comparative: across RoboTwin, LIBERO, and a real-robot platform, VersaCamVLA consistently outperforms prior VLA methods and direct multi-view baselines, and maintains robust performance as camera counts vary and camera poses shift away from training. The direct multi-view alternative — appending tokens from additional cameras to the policy input — improves coverage but still binds the policy to the camera count and view layout used during action learning, and its token length grows with the number of input views.

The evaluation perturbs each camera by sampling its pose on a spherical shell centered on the workspace: an azimuth offset drawn uniformly from −90° to 90°, a radial scale from 0.8 to 1.2 that rescales distance to the workspace, and a pitch from 25° to 75°, with the camera reoriented to look at the workspace center. Configurations are resampled per trial and shared between π0.5 and VersaCamVLA, so the comparison is matched. A stratified evaluation restricts the absolute azimuth offset to a specified interval while keeping the other sampling settings.

Latency is a separate axis. Measured on a single NVIDIA 4090 at batch size 1, VersaCamVLA incurs a smaller latency increase as additional input views are added than the three-view π0.5 reference marked in the figure. That is a statement about scaling with camera count, not about absolute speed at a fixed configuration.

Where the interface stops transferring

The method assumes calibrated camera intrinsics and extrinsics, and it focuses on tabletop bimanual setups with manual camera placement. It does not perform autonomous viewpoint selection, and the authors list uncalibrated cameras, active perception that optimizes camera configurations online, and mobile workspaces as future work. Safety-critical deployment, they add, needs further validation against task-specific failures under large camera-configuration shifts.

These conditions matter for how the results should be read. The robustness claim is about varying counts and perturbed poses within a calibrated, tabletop regime — not about arbitrary uncalibrated imagery or a policy that chooses its own views. A reader comparing this to geometry-aware approaches should also note the trade: VersaCamVLA avoids explicit 3D reconstruction at deployment and avoids extra multi-pose demonstrations, but it still requires calibrated geometry as input.

Compare camera-configurable policies on matched perturbations

The most decision-relevant feature of this paper is not a single success number but the evaluation design: camera configurations are resampled per trial and shared across the baseline and the proposed method, and the perturbation ranges are stated explicitly. That makes the comparison a test of the interface rather than of a lucky camera layout. When weighing a camera-configurable policy against a direct multi-view baseline, the question to ask is whether the baseline was given the same pose perturbations and the same camera counts, because a fixed-tuple policy evaluated on its training layout answers a different question.

The second priority is separating the two stages' contributions. The scene-token interface is what makes the input size independent of camera count; WAPS is what supplies pose diversity during its training; multi-signal supervision is what shapes the token content. The ablations support each of these separately, with the RGB-only versus multi-signal gap (20.94% to 29.50% average on RoboTwin 2.0 DR) being the largest reported single design effect. A reader deciding whether the approach fits a new platform should check first whether calibrated extrinsics are available and whether the workspace is tabletop, since those are the boundaries the paper itself draws.

Sources used in this answer