How does cross-attention routing capture spatiotemporal dynamics in visual cortex?

A learned query routes video features per parcel

The study asks whether fMRI responses to natural video are better predicted when a model can choose, for each cortical parcel, which locations at which moments matter. A parcel is a small cortical region whose voxels are modeled together. The backbone is V-JEPA-2 ViT-g/16, a self-supervised video model; each 3-second clip is sampled into 16 frames, and its tokens form a 16×24×24 grid where each token covers one sampled frame (about 0.19 s) and a 16×16-pixel patch.

A learned query for each parcel performs cross-attention over that token grid, and the attended feature is passed to a voxel-specific linear readout that predicts responses. In the joint model, the softmax runs over all (time, height, width) tokens, so a parcel can shift its weighting to different places at different moments. The authors compare this with constrained alternatives: mean pooling, temporal-only weighting, spatial-only weighting, a factorized form whose spatial and temporal weights multiply, and a fixed spatial distribution that acts as a static receptive field.

For a typical subject the joint model has about 47M parameters: roughly 0.6M queries, 21.8M in the shared attention and feed-forward block, and 24.5M in voxel readouts. Evaluation uses 102 held-out test clips, with 10 ensemble members trained on random 900/100 validation splits of 1,000 training clips; the ridge baseline uses the same splits.

Joint routing wins overall, but the margin is small

Averaged across all modeled voxels, the joint model reached r² = 0.191 versus 0.187 for the closest constrained model, spatial-only routing, a difference that held in all 10 subjects (t(9) = 5.85, FDR-corrected p < 0.001). The joint model also exceeded every other constrained model and ridge regression in all 10 subjects.

The constrained models show where the gain comes from. Of the 0.010 by which joint routing exceeded mean pooling, 0.006 was recovered by routing over space alone. Temporal-only routing did not improve over uniform pooling (0.180), and adding temporal routing to spatial routing in the factorized form did not improve beyond spatial routing alone (0.186). A fixed spatial distribution learned per parcel also failed to exceed mean pooling (0.181).

The authors read this pattern as evidence that the benefit requires stimulus-dependent selection of different locations at different moments, not merely more flexible weights. That is the paper's interpretation of the ablation pattern; the comparisons themselves are measured on held-out clips.

Lateral and parietal cortex benefit most from coupling

The advantage of joint routing over factorized and spatial models is concentrated in lateral occipitotemporal cortex. In lateral/face/body regions the joint model reached 0.263, or 66% of the noise ceiling, and exceeded the factorized, spatial, mean-pool, and ridge models in every subject (joint minus factorized: t(9) = 6.60, FDR-corrected p < 0.001). Parietal regions likewise favored joint routing over all other models.

Early visual cortex behaves differently. There the spatial, factorized, and joint models were tied at 0.189 (joint minus factorized: t(9) = −0.03, FDR-corrected p = 0.98), and all three improved on mean pooling by 0.012. In other words, stimulus-dependent selection of spatial locations accounts for the whole gain in early visual cortex, while lateral and parietal regions additionally benefit from following content over time.

Scene-selective regions show mixed evidence. Neither the joint model nor the spatial model significantly exceeded the ridge baseline there. The authors suggest this may reflect scene-selective cortex encoding global scene layout, which features averaged over the whole clip may already capture. In the scene-selective ROIs alone (PPA, RSC, TOS), the joint model did beat the factorized model in every subject (+0.0041 signed r², t(9) = 6.16, FDR-corrected p < 0.001) but matched ridge on mean-pooled features (+0.0007, t(9) = 0.42, FDR-corrected p = 0.75).

Attention maps track objects that move across frames

Because the joint model's attention is a distribution over tokens, it can be visualized per clip. For a parcel in left EBA, attention stayed on a fencer's body and, after a cut to a close-up, followed the arm and sword through the swing. For a parcel in left STS, it stayed on the head and body of a panda as it climbed.

The factorized model cannot produce this pattern. Its spatial map is fixed across frames and only its temporal weights change, so it concentrates attention on a few frames and spreads it over a large region that includes background. In the fencer clip, a single spatial map must serve both the wide shot and the close-up even though the body is in different places. This comparison is the paper's clearest demonstration of what joint routing adds beyond a separable spatial-by-temporal weighting.

Attention separates face, body, and scene networks

If a parcel's attention reflects what its voxels respond to, category-selective regions should attend to their preferred category. The authors test this using the intersection over union (IoU) of each parcel's 92 most-attended tokens, the top 1%, with category masks from segmenting the clips with SAM 3. Raw IoU mostly reflects category size, so they z-score it across parcels to isolate parcel differences.

FFA, OFA, and EBA attended more than the average parcel to faces, bodies, and animals, which usually appear together in the clips, and less to scenes; scene-selective PPA and RSC showed the reverse. For most face-, body-, and scene-selective ROIs, the highest category was the expected one, with exceptions for OFA and STS.

The authors then derive localizer contrasts and find that the highest-contrast parcels overlap the corresponding ROIs above chance: face 0.19 versus chance 0.09, body 0.14 versus 0.02, and scene 0.28 versus 0.10 (FDR-corrected p < 0.05). A control using masks from a different clip matters here: for faces, mismatched masks gave overlap at chance (0.10 versus 0.09) while correct masks were significantly larger, so face localization depends on clip content. For scenes, mismatched masks gave nearly the same overlap as correct masks (0.26 versus 0.28), so scene localization mostly reflects the parcels' spatial priors rather than attention to scenery in each clip.

Short clips and slow fMRI bound the claims

The stimuli are 3-second clips from the BMD dataset, so the framework has not been tested on longer movies where temporal integration over longer windows could matter. fMRI's slow temporal resolution is a second constraint; the authors propose testing the model on MEG and EEG data with finer temporal resolution.

The paper also separates its computational result from a biological claim. The authors write that the results could suggest higher visual cortex implements a mechanism analogous to spatiotemporal cross-attention, but state that this remains to be empirically tested. They note that scaled dot-product attention may approximate a normalization mechanism and that future work should compare cross-attention against biologically inspired normalization models.

The advantage of coupled routing in lateral regions is not specific to V-JEPA-2: with late-layer features from VideoMAE v2 and the image models DINOv2 and CLIP, the joint model was again more accurate than the factorized model in lateral/face/body regions. The pattern depends on features, though: with early-layer features the joint model was less accurate than spatial-only routing, and for the last stage of CLIP, ridge regression on mean-pooled features was most accurate in every region except early visual.

Coupling matters where content moves through space and time

The paper's most decision-relevant comparison is not joint routing versus mean pooling but joint routing versus spatial-only routing, because that isolates what temporal coupling adds. The answer is region-specific: in early visual cortex, spatial routing alone captures the entire gain; in lateral/face/body and parietal regions, joint routing adds a further, consistent improvement across subjects; in scene-selective cortex, the evidence for routing beyond mean-pooled features is weak.

That split gives readers a concrete threshold for when the framework is worth the added complexity. If the target is dynamic motion and action processing in lateral or parietal cortex, the joint spatiotemporal query is the model to use. If the target is early visual cortex or global scene layout, a simpler spatial or mean-pooled readout performs comparably, and the interpretable attention maps should be read with the clip-content controls in mind, especially for scene-selective parcels.

Sources used in this answer