Can fMRI visual decoding be unified across participants and datasets in one space?

One decoder, eight datasets, one CLIP space

The paper asks whether fMRI measurements collected from different people, scanners and experiments can be organized into a single visual-semantic geometry that supports decoding. Its answer is a qualified yes: BRAID-fMRI (Brain Representation Alignment across Individuals and Datasets) uses one ROI-wise Transformer whose parameters are shared across eight visual-fMRI datasets, covering 93 dataset-specific participant entries, 430,007 single-trial responses and 162,839 unique stimuli.

The shared target is not another brain. It is a 512-dimensional CLIP ViT-B/32 embedding space, and the model is trained so that a brain response lands near the CLIP representation of the image that evoked it. Because repeated presentations of the same stimulus across participants and datasets are treated as positives in a multi-positive contrastive objective, the model is pushed to place those heterogeneous responses close together rather than learning each participant in isolation.

Anatomical ROIs make heterogeneous brains comparable

The harmonization step is what makes a single model possible. Each single-trial response is represented using 379 anatomically defined regions of interest: 360 cortical parcels from the HCP-MMP1.0 parcellation plus 19 subcortical regions. Because regions contain different numbers of vertices or voxels, each ROI gets its own two-layer MLP tokenizer that maps its raw measurements into a common 512-dimensional token space.

After tokenization, every token carries a learned ROI identity embedding and, optionally, a participant embedding. The Transformer, pooling and projection layers stay shared across all 93 participant entries and all eight datasets. The participant vector is a light conditioning mechanism that can absorb individual and dataset differences; it is not the shared representation itself, which is why the paper can later test models trained without it.

Matched participants retrieve better than two baselines

The headline comparison uses eight matched participant entries for which all three methods were evaluated. BRAID-fMRI reaches 35.0 ± 11.1% Top-10 accuracy, against 27.1 ± 5.1% for a MindEye-style pooled-CLIP decoder and 21.1 ± 9.1% for ridge regression. It has the highest observed accuracy for seven of the eight entries.

Two details matter for reading that number. The MindEye-style baseline predicts a single pooled CLIP embedding for retrieval and does not include the diffusion prior or image reconstruction of the full MindEye framework, so this is a retrieval comparison, not a reconstruction comparison. And Top-10 accuracy here means the correct stimulus appears among the ten nearest candidates in the shared space, so the denominator is the candidate gallery, not the number of brain voxels or participants.

A broader check across all 63 dataset–participant entries where both methods were evaluated gives BRAID-fMRI a mean Top-10 of 22.2% versus 13.1% for ridge regression, with a higher dataset-level mean in six of seven retrieval datasets. NOD is the exception, where ridge regression leads at 12.8% versus 8.1%.

Unseen datasets transfer without fitted participant vectors

The stronger claim is not that the model fits many participants, but that it can generalize to participants it never saw. In separately trained participant-agnostic models, participant embeddings are disabled and an entire target dataset, including all of its participants, is held out during training. These models still support target-dataset holdout retrieval.

Expanding the source pretraining pool increased target-dataset holdout accuracy by up to 92.1% relative to the initial source-training condition, and the final, more diverse condition outperformed the initial condition for all seven targets. The paper is explicit that this comparison jointly changes dataset diversity and the amount of training data, so the 92.1% figure cannot be attributed to diversity alone. Individual curriculum steps were also not uniformly beneficial, which the authors read as evidence that additional neural data are not interchangeable simply because they add samples.

The learned space preserves graded semantic structure

Retrieval accuracy alone would not show that the shared space is organized like a visual-semantic geometry. The paper therefore compares brain–brain cosine similarity for three relationships: the same stimulus, different stimuli within the same CLIP-derived semantic cluster, and stimuli from different clusters. Same-stimulus embeddings are most similar, same-cluster pairs occupy an intermediate regime, and different-cluster pairs are least similar, and this ordering persists for comparisons across datasets.

At the dataset level, representational dissimilarity matrices built from semantic-cluster centroids show correspondence among the evaluable dataset pairs, with Spearman correlations reported for pairs sharing at least five semantic clusters. A neighbourhood analysis also asks whether stimuli that are close in CLIP space are close in the learned brain space, comparing observed k-nearest-neighbour overlap against an analytical random expectation. These analyses test structure in the predicted space; they do not establish that this geometry is uniquely or completely representative of human visual organization, a point the authors raise themselves.

Ventral and early visual cortex carry most of the signal

Ablations and ROI-level saliency concentrate model sensitivity in ventral visual/object and early visual regions, with category-dependent contributions from motion and attentional systems. Across the full set of 40 semantic clusters, the size and relative contribution of individual networks varied considerably, so the cortical story is content-dependent rather than a single fixed map.

The authors qualify this anatomical reading directly: network masking and gradient attribution characterize the fitted decoder and can reflect group size, redundant inputs and distribution shift. Biological interpretation would require attribution validation and perturbation controls, and these analyses do not establish causal necessity of the corresponding brain systems. The sensitivity result is therefore evidence about what the model uses, not proof of what the brain requires.

Diversity of source data matters more than raw scale

The paper's most decision-relevant comparison is not the 35.0% headline but the holdout result: a shared decoder can transfer to an unseen dataset without fitting participant embeddings, and the final, more diverse source pool improved every one of the seven targets. That makes cross-study variation something to model jointly, rather than a nuisance to be removed before pooling.

The priority it sets is conditional. Because the source datasets remain highly imbalanced in participants, trials and stimulus composition, the experiment cannot separate diversity from scale, and the non-monotonic curriculum steps show that more data is not automatically better. The practical reading is that anatomical correspondence, cross-participant structure and stimulus semantics should be modelled together, while the specific mix of source datasets remains a variable to be chosen and tested rather than assumed.

Sources used in this answer