Can brain encoding models be fine-tuned to match individual perceptual similarity?

Accurate brain prediction is not the same as perception

Visual brain encoding models are trained to predict measured brain activity from images. The paper asks a different question: do the internal representations of such a model reflect how a particular person perceives two images as similar? A model can predict voxel responses well and still organize images in a way that does not match anyone's perceptual similarity space.

The authors treat this as an alignment problem. They compare three kinds of representational similarity matrices, or RSMs: one built from a person's similarity judgments, one from measured fMRI responses, and one from model-predicted responses. Each RSM records how similar every pair of images is under that source, so the comparison asks whether images a person finds similar also produce similar brain activity, measured or predicted.

The data come from the Natural Scenes Dataset: 8 participants, whole-brain 7T fMRI, and about 9,000 unique training images per person plus 1,000 shared images. Perceptual similarity judgments exist for only 100 shared images, collected with a multiple-arrangement task in which participants placed more similar images closer together within a circle, yielding a 100×100 distance matrix per person.

Baseline encoders align better than raw fMRI

Before any fine-tuning, the baseline encoders already tracked within-subject perceptual similarity more strongly than measured fMRI did. Correlating each person's judgment RSM with the two brain-based RSMs gave a mean Spearman ρ of 0.198 for the baseline encoder and 0.146 for measured fMRI, a significant difference across the 8 participants (W = 0, p = 0.008), with every subject significant in both cases.

The authors suggest this may reflect the higher signal-to-noise ratio of model predictions relative to raw fMRI, but they present that as a hypothesis rather than a tested mechanism. The practical point is narrower: a well-fit encoder is not a blank slate for perception, yet its advantage over measured brain data is about within-person image-pair structure, not about individual differences between people.

That distinction matters because the same baseline encoders preserved the across-subject structure of measured fMRI almost exactly (ρ = 0.819, p = 0.008) while showing no correspondence with the across-subject structure of human judgments (ρ = −0.028, p = 0.890). Measured fMRI itself also failed to match judgment-based intersubject structure (ρ = −0.193, p = 0.345).

Personalizing DreamSim from 100 images per person

Fine-tuning needs similarity targets for all ~9,000 training images, but only 100 images per participant were judged. The authors therefore use DreamSim, a pretrained model whose embedding space was optimized to reproduce population-level human similarity judgments, as a stand-in that can score any image pair.

To personalize it, they keep DreamSim frozen and learn a 1,792-dimensional reweighting vector per participant that rescales the embedding dimensions before computing cosine similarity between images. The vector is fit to that person's observed judgments with a regularization term pulling it toward the original unit vector, using 5-fold cross-validation over the 100 images and 10 repeats of independent fold assignments.

This step worked, but modestly. Population DreamSim correlated with individual judgments at ρ = 0.245 ± 0.030 across subjects, and personalized DreamSim reached ρ = 0.314 ± 0.025, a mean gain of +0.069 that appeared in 8 of 8 participants (W = 0, p = 0.008). The personalized model is thus a better proxy for a person's judgments, not a replacement for measuring them.

Two fine-tuning targets produce different encoders

Each participant's baseline encoder was then fine-tuned as a student model, with a frozen copy of the baseline as teacher. The loss combined two terms: mean squared error against the teacher's voxel predictions, weighted 10, and a term weighted 2 that maximized correlation between the student's image-pair similarity structure and a DreamSim-derived target. The only thing varied between conditions was whether that target came from population or personalized DreamSim, with identical initialization and objective.

Within-subject alignment with human judgments rose across the three models: baseline encoder mean ρ = 0.192, encoder plus population DreamSim mean ρ = 0.287, and encoder plus personalized DreamSim mean ρ = 0.377, with all pairwise comparisons significant. The personalized encoder also exceeded personalized DreamSim alone (0.377 versus 0.314, W = 0, p = 0.008), so the gain is not merely inherited from the similarity model used as supervision.

The across-subject picture separates the two conditions more sharply. Personalized DreamSim reproduced intersubject judgment structure at ρ = 0.766, and the encoder fine-tuned on it recovered that structure at ρ = 0.508 (p = 0.002), positive across all 10 cross-validation repeats (range 0.38–0.57). Neither the baseline encoder (ρ = −0.026) nor the population-fine-tuned encoder (ρ = −0.099) showed significant correspondence.

Personalized alignment trades away fMRI structure

The personalized encoder did not simply improve on every measure. The baseline encoder strongly preserved the intersubject structure of measured fMRI (ρ = 0.84, p = 0.007), and this remained high after population-level fine-tuning (ρ = 0.77, p = 0.014). After personalized fine-tuning, that correlation was no longer significant.

So the two fine-tuning conditions move the encoder in different directions. Population-level supervision makes the encoder better aligned with human judgments on average while keeping it close to the intersubject organization of measured brain responses. Personalized supervision makes it reflect both within-subject and across-subject judgment structure, but at the cost of no longer matching how measured fMRI varies between people.

This is a comparison between representational targets, not evidence that one target is universally correct. Which encoder is preferable depends on whether a downstream use needs agreement with measured brain responses or with a specific person's perception.

Alignment gains outweigh voxel accuracy losses

Fine-tuning also changed voxel-level prediction accuracy, and the paper reports the tradeoff explicitly. Relative to baseline, the personalized encoder lost a mean 9.2% in voxel prediction accuracy and the population encoder lost 7.1%, while human similarity alignment gained a mean 190% and 119% respectively.

The averages hide substantial variation: some participants gained a lot of behavioral alignment with only modest accuracy changes, while others showed larger reductions in voxel prediction. The authors describe this as a subject-dependent accuracy-versus-alignment tradeoff rather than a fixed exchange rate.

At the level of individual image pairs, the shift was asymmetric. From baseline to personalized encoder, 3.5% of image pairs moved from agreement with the human judgment to disagreement, while 6.2% moved from disagreement to agreement, so the net movement was toward matching the person's similarity assessments.

What the framework can and cannot deliver

The paper's contribution is a procedure for turning a small number of human similarity judgments into supervision for a brain encoder: adapt a generic perceptual model per person, generate similarity targets for the full training set, then fine-tune the encoder while anchoring it to its original voxel predictions. The authors frame this as bootstrapping limited behavioral data into more individualized, human-aligned models of vision.

The main stated limitation is that DreamSim serves as a proxy for human judgment, so the quality of the personalized targets depends on that base model. The authors note that newer vision or vision-language foundation models could provide a stronger population-level baseline and suggest testing whether other perceptual models make better targets for personalization.

The practical priority follows from the comparison rather than from the average alignment score. If the goal is an encoder whose image representations track one person's perception, including how that person differs from others, personalized supervision is the condition that achieved it, and the roughly 9% mean voxel-accuracy cost is the price the paper measured. If preserving agreement with measured fMRI intersubject structure matters more, population-level fine-tuning kept that correlation high while still improving average judgment alignment.

Sources used in this answer