Can Masked Cross-View Attention Improve Robustness of 3D Foundation Models?

Dense cross-view attention is not always reliable evidence

Feed-forward 3D foundation models estimate camera parameters and dense scene geometry from multiple images in one forward pass, replacing the iterative, scene-specific optimization of conventional Structure-from-Motion pipelines. Their encoder aggregates multi-view information through all-to-all global attention. The paper argues this design has two costs: computation grows quadratically with the total number of frames, making long sequences increasingly expensive, and unconstrained cross-view interaction can propagate unreliable evidence.

The unreliable evidence the paper names is specific. Visually similar but geometrically distinct structures, called doppelgangers, can induce incorrect associations across views, and occluded views can contribute misleading information. The authors draw an analogy to calibration view graphs in COLMAP-style pipelines, where edges encode geometrically verified image pairs and unreliable pairs can be pruned before bundle adjustment. Their question follows from that analogy: can 3D foundation models learn a more robust representation when frames are missing?

MGE masks frame connections, not image content

MGE is a training framework for feed-forward 3D architectures that use cross-view attention. Instead of masking image content, it randomly removes frame-to-frame connections during global attention, restricting the cross-view information available to the geometric encoder and simulating a sparse calibration view graph. The underlying scene geometry is unchanged: the network must still recover consistent 3D structure from the remaining cross-view interactions.

The masking is implemented at the token level. Given query, key, and value matrices, the masked global attention adds a token-level mask induced by a frame-level connectivity matrix, setting the mask to zero where a frame pair is connected and to negative infinity where it is not. Masking applies only to global attention; frame attention is unchanged, so each image still fully models its intra-frame spatial structure while cross-frame exchange is restricted. Because different connections are removed across iterations, the encoder cannot rely on particular view pairs and each frame representation must retain more geometrically informative context. The paper describes MGE as model-agnostic and uses pi-3 as its primary instantiation, with additional experiments on VGGT reported in the appendix.

Teacher distillation keeps the masked student from degrading

Naively removing or sparsifying global interactions can cause substantial performance degradation, which the paper attributes to pretrained representations depending heavily on dense cross-view connectivity. MGE addresses this with a self-distillation design: a frozen pretrained teacher is evaluated with dense global attention, while the student uses masked global attention, and the teacher's predictions serve as supervision instead of sparse ground-truth annotations.

The training objective combines geometry and feature distillation. The geometric loss contains a scale-invariant point-map loss, a surface-normal loss, a confidence loss, and a relative camera-pose loss; because point maps are defined up to a scene-level scale, the student prediction is aligned to the teacher pseudo target with a single scale factor. Feature distillation adds cosine-distance losses on the final-layer register tokens and image tokens before the prediction heads. The authors state that this intermediate supervision is what avoids performance degradation, and they validate the design choice through ablations and appendix analysis.

Robustness gains appear under occlusion and doppelganger views

The paper reports that MGE leads to much stronger performance under occlusion and doppelganger views while retaining high performance on standard benchmarks. These are two distinct evaluation settings, and the paper treats them separately: occlusion concerns missing or unreliable visual evidence from a view, while doppelganger views concern visually similar but geometrically distant structures that can induce incorrect cross-view associations.

The claim is comparative. The authors describe MGE as significantly more robust and accurate than previous state of the art in these challenging settings, while retaining top performance in conventional evaluations. The mechanism they offer is that training under varying connectivity patterns discourages excessive reliance on particular view pairs, so representations remain stable when individual view relations are missing or unreliable. This is the paper's own explanation of why the gains occur, not a separately measured causal decomposition.

Richer frame representations make token merging more effective

The paper connects robust representation learning to efficient long-sequence inference through Anchor-Guided Adaptive token merging. Standard token merging reduces inference cost by combining similar or redundant tokens before global attention, but the authors argue token-level similarity does not necessarily imply geometric redundancy: repetitive structures such as windows on a building facade may produce similar features and be merged into one token despite corresponding to different 3D locations, removing spatially distinct geometric evidence.

AGA compresses at the frame level instead. Given N input frames, it uniformly samples a subset as anchor frames, keeps them at full token resolution, and distributes them across the input sequence for coarse but complete coverage. Each non-anchor frame is mean-pooled into a frame descriptor, and its maximum cosine similarity to the anchor frames becomes a redundancy score. High-scoring frames are compressed more aggressively; low-scoring frames, which provide complementary content, retain more tokens. Anchor frames use a merging ratio of zero and remain uncompressed. Merged non-anchor outputs are restored to the original token layout using recorded merge assignments and passed to subsequent layers and prediction heads without architectural modification.

AGA trades speed for quality better than uniform reduction

Compared with other efficient inference approaches, the paper reports that AGA achieves inference speedup while consistently maintaining higher reconstruction quality, particularly in limited-view settings. The advantage is described as especially pronounced when neighboring views are farther apart, where uniform token reduction can discard geometric information and cause severe reconstruction degradation.

The comparison is between AGA and existing token-merging methods at comparable computational cost, and the paper attributes AGA's edge to the richer, more independent per-frame representations learned by MGE. The authors also report that AGA improves long-sequence efficiency especially in low-texture and repetitive scenes. The merging ratio is calibrated so that the total number of merged tokens across non-anchor frames matches a target budget under a maximum-ratio constraint, which is what makes the comparison a like-for-like cost comparison rather than a free quality gain.

Where these claims hold and where they do not

The evidence in this paper is experimental and comparative, not a guarantee of transfer. MGE is presented as a model-agnostic framework for feed-forward 3D architectures with cross-view attention, but the main paper's instantiation is pi-3, with VGGT experiments placed in the appendix; the robustness and efficiency numbers reported in the main text therefore belong to that primary instantiation and its evaluation settings.

The occlusion and doppelganger gains are measured against previous state of the art on the paper's own benchmarks, and the AGA speedup is measured against other efficient inference approaches at comparable computational cost. The paper does not claim that masking always helps, and it explicitly notes that naive removal or sparsification of global interactions can degrade performance substantially, which is why the teacher-distillation supervision is part of the method rather than an optional add-on. The reported gains also depend on the redundancy structure of the input sequence: AGA's frame-level compression assumes anchor frames give coarse but complete coverage, and its advantage is described as most pronounced when neighboring views are farther apart.

The priority is regularizing connectivity, not adding views

The most decision-relevant result in this paper is that regularizing cross-view connectivity during training, rather than adding more views or more attention, is what produces robustness to occlusion and doppelganger views while preserving standard-benchmark performance. The paper's framing makes this a deliberate trade: it accepts incomplete cross-view context during training and compensates with teacher distillation, then exploits the resulting per-frame independence at inference.

For readers comparing approaches, the paper's own split matters more than any single number. If the problem is unreliable or ambiguous view relations, MGE targets the learned interaction itself, whereas methods such as GLUEMAP, VGGT-Long, and Diversity-aware VGGT modify which views interact or how predictions are combined at inference time. If the problem is long-sequence cost, AGA's frame-level, anchor-guided compression is the paper's answer, and its reported advantage over uniform token reduction is largest when neighboring views are far apart and in low-texture or repetitive scenes. The paper's contribution is the combination: robustness learned under partial connectivity, then spent as inference-time compression.

Sources used in this answer