How can receptive-field constraints optimize stimuli for early visual cortex?

Small receptive fields break region-level MEI generation

Stimulus optimization asks a brain encoder which image would most strongly drive a target neural population, and returns that image as a most-exciting-input, or MEI. The approach had already been applied to human higher visual cortex with fMRI data, but early and intermediate retinotopic areas were a different problem: their responses are strongly shaped by retinotopic organization and by small population receptive fields, and existing fMRI stimulus optimization methods did not model those constraints.

A population receptive field, or pRF, is the region of visual space whose stimulation drives a given voxel. Because early areas have small pRFs, a stimulus that excites one voxel can look almost arbitrary elsewhere in the image, and averaging across voxels in a region would blur selectivity that varies widely between neighboring sub-populations. The paper therefore targets single voxels rather than regions, and builds the spatial constraint directly into the encoding model.

A Gaussian pRF turns CNN features into voxel predictions

Both frameworks share one backbone: a subject-specific, voxelwise feature-weighted receptive field (fwRF) encoding model. It takes intermediate features from a pretrained deep convolutional network, spatially pools them according to an estimated voxel-specific Gaussian pRF, and uses the pooled features to predict that voxel's fMRI response. The Gaussian's center and size are fitted by grid search over x, y and sigma, so the model carries both what features a voxel prefers and where in the visual field it looks.

The two frameworks then diverge in how they use those predictions to make an image. RF-GO (Receptive Field Gradient Optimization) performs regularized gradient ascent directly on the image, pushing predicted response upward. RF-DiVE (Receptive Field Diffusion for Visual Exploration) uses the response gradient to guide a pretrained latent diffusion model, so a natural-image prior shapes the result. The comparison matters because the prior is a design choice, not a detail: early visual cortex is not strongly selective for semantics, so it was unclear whether a diffusion prior was needed for interpretable MEIs at all.

MEIs concentrate structure inside the pRF

For an example V1 voxel, the top MEIs from both frameworks showed consistent feature patterns within the pRF, marked as the 3-sigma range of the Gaussian, while remaining largely random outside it. That spatial specificity is the visible signature of the pRF constraint: the optimizer is free to place anything elsewhere, and it does.

Within the pRF, the MEIs carried coherent structure rather than noise. The paper's examples include high-contrast yellow-black rectilinear and angular contours, multicolored bent lines and curves forming abstract patterns, clusters of dark dots on warm backgrounds, and dense pinkish or greenish speckled patterns. Similar features appeared in the top natural images for the same voxels, such as a bright yellow railing or pink flowers with green stems, but inspecting those natural images alone did not always reveal a coherent shared feature set. The authors read this as evidence that MEI synthesis can expose the features driving a population when natural images do not.

The predicted-response advantage survives an independent encoder

The central quantitative test is in-silico: predicted responses to MEIs were compared with predicted responses to natural images, using an independent test encoder with a different architecture, pretraining dataset, and NSD training partition. The comparison used the top-20 voxels ranked by R-squared in each visual area, and the stricter natural-image control was LAION-fMRI rather than the NSD images the models were trained on.

Across all methods and all visual areas, MEIs elicited higher model-predicted responses than the most activating natural images. The advantage was largest when evaluated with the generation encoder and attenuated on the ranking and test encoders, but the predicted difference score stayed positive for all four NSD participants even in the most challenging case of generalization to the test model. The advantage was significantly greater than zero for the majority of subjects in every visual area and for every generation method (one-sample t-test, p<0.01, FDR corrected). This is a model-predicted result, not a measured neural response; the paper states that closed-loop experiments with human participants are still needed for further validation.

Generation method and backbone shift the tradeoff

RF-GO and RF-DiVE did not differ by a single ranking. Evaluated with the generation encoder, RF-GO achieved larger difference scores than RF-DiVE under both backbones. That advantage was not consistently retained under the ranking or independent test encoder, which the authors interpret as the naturalistic prior of RF-DiVE producing weaker activation of the generation model but better generalization across backbones.

The backbone effect depended on the framework. For RF-DiVE, the DINO backbone yielded larger scores than ADV under the generation and ranking encoders, and DINO had consistently higher difference scores for the majority of participants in all areas in most independent-test comparisons. For RF-GO, DINO was also larger on the generation encoder, but the pattern partially inverted on the ranking and testing models, with DINO similar to or worse than ADV in all regions. The authors suggest DINO paired with RF-GO is prone to model-specific biases that impair cross-model generalization, while ADV gives more robustly generalizable RF-GO MEIs.

The image statistics also separated the methods. RF-DiVE MEIs showed higher colorfulness than RF-GO MEIs and natural images, with higher subject-averaged a* and b* values indicating a stronger pull toward warm reds and yellows, under both backbones. Relative to natural images, both methods showed a smaller contribution at the lowest measured spatial frequency and a larger one at intermediate frequencies, and within each framework DINO-generated MEIs shifted further toward intermediate and higher spatial frequencies than ADV-generated ones.

A behavioral test favors RF-DiVE across areas

To check whether MEI differences are perceptually meaningful, the authors ran a forced-choice human study with 32 participants in each of two experiments. Participants judged whether hV4 MEIs or V1 MEIs showed more three-dimensional form, a dimension chosen because sensitivity to shape and form is expected to increase from V1 to hV4.

Participants more often chose hV4 RF-DiVE MEIs as having greater three-dimensional form than V1 MEIs, across both encoder backbones and both viewing conditions (pRF-zoomed and full image). No reliable hV4-over-V1 preference appeared for natural images from LAION. Results for RF-GO were less consistent across backbones and viewing conditions, though the ADV backbone showed an hV4 preference in the full-image condition. The authors present this as initial characterization and call for larger behavioral studies; the paper also notes that its quantitative image analyses focus on relatively low-level properties such as Gabor features, color, edges, and curvature, leaving texture, contour, and surface structure less directly measured.

Read the method pair as a robustness check

The paper's most useful practical distinction is not which framework wins overall but what each one buys. RF-GO maximizes the generation encoder's predicted response more aggressively, while RF-DiVE trades some of that activation for images that generalize better to an independent encoder and that carry more interpretable regional differences in the behavioral test. The backbone choice is not neutral either: DINO helps RF-DiVE but can hurt RF-GO's cross-model generalization relative to ADV.

That is why the authors propose combining multiple generation methods as a strategy to separate robust, generalizable tuning properties from properties idiosyncratic to one method. For a reader deciding how to probe a single voxel in V1-hV4, the paper's evidence supports treating an MEI as jointly shaped by the target voxel, the encoding backbone, and the optimization framework, and treating features that survive across those choices, such as the curves and texture visible in both frameworks' images, as the more trustworthy candidates for a voxel's selectivity.

Sources used in this answer