A prediction target built from eye movements
The paper starts from a property of natural vision rather than from a new network block. Each fixation delivers a small high-resolution crop, a glimpse, and successive glimpses are related: some scene parts tend to co-occur, and the direction of the eye movement constrains what can appear next. The authors ask whether a network can learn scene structure by predicting the features of the upcoming glimpse instead of being trained on object labels or language captions.
Concretely, GPNs receive the embedding of the current glimpse and predict the backbone's embedding of the next glimpse. The target is not raw pixels but a high-level feature vector, extracted by a frozen pretrained backbone; the paper argues pixel-level prediction would need a much larger generative network and has been shown to rely on features uninformative for high-level object perception. Only the GPN parameters are trained, with a contrastive loss that pulls the prediction toward the true next glimpse and pushes it away from other glimpses in the same sequence and in other scenes.
Four variants isolate recurrence and saccades
The design is a 2×2 comparison. Recurrence (R) is implemented as a three-layer LSTM with hidden size 1024, or as the same forward pass with recurrent components zeroed. Saccade information (S) is a 2D displacement vector from the current glimpse center to the next one. The variant with neither is called B, the baseline, giving GPN-B, GPN-R, GPN-S and GPN-RS.
Glimpse sequences were sampled with DeepGazeIII, a model that predicts scanpaths rather than static saliency, over COCO scenes. Each scene yielded 10 fixation sequences of 7 fixations starting at image center; 91-pixel glimpse crops were taken around fixation centers, corresponding to a 3-degree window under the paper's assumed 8.4-degree display. Training used roughly 115k scenes in the train-515 split, and evaluation used the special-515 scenes that each participant saw three times in the Natural Scenes Dataset (NSD). Three frozen backbones were tested: ResNet50 trained on ILSVRC, ResNet50 trained with SimCLR, and a DINOv2 ViT.
Emergent features align better than their inputs
The central result is a comparison between the GPN's internal representations and the frozen glimpse features it was built on. Across visual cortex, the emergent GPN representations outperformed their input-glimpse features in alignment with scene representations, and they also beat Glimpse Stitching Networks trained to predict scene objects or caption embeddings. In mid and higher-level visual cortex, they equated or outperformed previously reported state-of-the-art ventral stream models, including large-scale foundation models.
The paper's most striking claim concerns shared variance. The variance in ventral visual cortex, and in midparietal and parietal regions, explained by diverse models — supervised, self-supervised, vision-only and visuo-linguistic — was largely subsumed by the best-aligned GPN, GPN-R with the ResNet50-SimCLR backbone. The authors interpret this as evidence that the shared variance hinges on what GPN-R captured: co-occurrence of scene parts and integration across them. They further suggest that previously reported language-model–brain alignment might be mediated primarily by co-occurrence structure, the same signal that underpins distributional semantics in next-token prediction. That is the authors' interpretation, not a separately tested causal claim.
Explicit saccade information often reduced alignment
An unexpected result cuts against the intuition that more task-relevant information should mean better brain alignment. Adding the saccade signal often reduced GPN alignment with NSD scene representations. This was clearest in the non-recurrent models, where GPN-S aligned worse than GPN-B across nearly all backbones, but it also occurred for some recurrent models. The same saccade information improved next-glimpse prediction, so better task performance did not imply better cortical alignment.
The authors offer several possible explanations rather than a settled one. GPN-B, lacking explicit saccade input, can only solve the task by learning more location-agnostic scene associations — given one glimpse, what other content is likely in the same scene — and such associations may better match centrally fixated NSD responses. GPN-RS, by contrast, can use recurrence and saccades to maintain what appeared where, which may or may not help depending on the backbone. The reduction could also reflect limited spatial-arrangement information in NSD responses or a mismatch between GPN and cortical spatial codes; the paper calls disambiguating these accounts an important next step.
Co-occurrence and arrangement are learned, not labeled
The paper reports that GPNs successfully extract complex scene information, including object co-occurrences and spatial object arrangements, and integrate information across glimpses. This matters because none of it was supervised: the GPN parameters were trained entirely through self-supervised next-glimpse prediction, without scene labels, language supervision or neural data, while the backbones and DeepGazeIII were pretrained and frozen.
The stitching networks serve as a control on the objective itself. Trained with the same architecture but different output sizes, the caption-embedding variant (sGSN) aligned better with ventral visual cortex than the object-classification variant (cGSN; mean difference 0.017, W=0, p=0.0078), replicating a previously reported finding. The sGSN alignment was also comparable, slightly higher, than a previously reported recurrent architecture trained only on full COCO scenes (mean difference 0.02, W=0, p=0.0078). These checks support the claim that the prediction objective, not just architecture or data volume, shapes alignment.
The alignment is real but partial
The paper is explicit that even its best models explain less than half of the variance shared across participants. The alignment is measured against NSD fMRI responses to the special-515 scenes, so it concerns scene representation in that dataset and those regions, not a general account of visual cortex. The backbone choice also mattered: the recurrent-with-saccades variant's alignment was backbone-dependent, and the authors note that training the backbone with their procedure did not yield meaningful behavioral or neural alignment, possibly because COCO's roughly 115k images are less diverse than the datasets the backbones were pretrained on.
The comparison tables put the numbers in context. Average ventral visual cortex alignment was 0.12 for the RN50-SimCLR backbone on full scenes and 0.13 for RN50-ILSVRC, while several vision and vision-language baselines ranged from 0.07 to 0.19 across their best-aligned layers. The paper's claim is therefore a comparative one: the GPN's emergent representations match or exceed these alternatives, not that they fully capture cortical responses.
Predicting the next glimpse as a biological route
The paper's argument is about which signals a model is allowed to use. Rather than testing ever-larger networks engineered for AI performance, the authors take stock of what the visual system has available during active vision and build the objective from that. The success of GPNs is presented as evidence that next-glimpse prediction is a biologically plausible route toward brain-aligned scene representations, and the authors frame it as part of a broader principle of discovering world structure through temporal, sensorimotor prediction.
For readers weighing models of cortical scene representation, the paper-specific priority is the comparison between GPN-R and the alternatives, and the counterintuitive status of saccades. The model that aligned best was the recurrent one without explicit saccade input, and giving models more spatial information did not consistently improve alignment. That makes the location-agnostic co-occurrence signal, rather than location-specific prediction, the more promising target for future models — while the unexplained portion of shared variance remains the gap to close.
Sources used in this answer
Predicting upcoming visual features during eye movements yields scene representations aligned with human visual cortex
Recurrent Glimpse Prediction Networks trained only to predict the next glimpse's high-level embedding along human-like scanpaths produced scene representations that aligned strongly with human fMRI responses in mid and higher-level visual cortex, matching or outperforming alternative state-of-the-art ANN models; the best-aligned variant was the recurrent model without explicit saccade input.