How do recurrent network dynamics explain perceptual grouping in natural scenes?

Why natural scenes break earlier grouping models

Perceptual grouping is the process that binds spatially distributed image elements into a single object and separates that object from its surroundings. A first feedforward sweep through visual cortex extracts edges, textures and colors and builds increasingly abstract object signals, but it does not by itself specify which of those features belong together. Grouping instead unfolds over hundreds of milliseconds and is thought to depend on recurrent interactions: feedback from higher to lower areas and horizontal connections within areas.

Earlier computational accounts of this process were built for simplified stimuli, such as isolated curves or object outlines on uniform backgrounds, where object boundaries are given explicitly. Those models cannot explain human behavior in naturalistic settings, where objects contain internal contours, colors and textures that are not true object boundaries. The paper's question is therefore whether a recurrent network can learn to group features in natural images and whether its time course matches human perception.

A frozen feature map plus recurrent grouping

The model, called GammaNet, does not see raw pixels. Its input is the highest-resolution feature map of a Panoptic R101-FPN, a feedforward segmentation network pretrained on COCO that was kept frozen during training. That input preserves fine spatial detail and carries semantic information, but it does not say which regions belong to the same object, especially when an image contains several objects of the same category.

GammaNet itself is a recurrent network of four layers of horizontal gated recurrent units (hGRUs) at progressively coarser spatial scales, with feedforward, horizontal and top-down connections. Inhibitory interactions produce surround suppression, while excitatory interactions let activity spread across space. A linear readout on the lowest layer turns activity into per-pixel class labels. The network was trained on COCO images with cues placed on objects, using a segmentation objective: pixels of the cued object get its class label and all other pixels are background. Because that target mask specifies the cue's relation to every image location, each training example supervises many pairwise same-different judgments. Training used contractive recurrent back-propagation, a learning rule that is local in time rather than requiring storage of the full activity trajectory.

Segmentation accuracy close to a far larger model

Across five independently trained networks, mean intersection-over-union (IoU) between predicted and ground-truth masks was 0.74 ± 0.01 (mean ± SEM). IoU is the overlap between predicted and true masks divided by their union, so 1 would be perfect overlap. For comparison, the transformer-based Segment Anything Model reached 0.81 ± 0.02 on the same images.

The authors attribute the small gap to scale and training rather than to the grouping principle: GammaNet has about 8 million parameters versus about 632 million for SAM, and saw roughly 5 million image-passes during training versus about 23 million for SAM. This comparison matters because it shows the recurrent model is a competent segmenter, not merely a qualitative demonstration; the grouping dynamics analyzed next are read out from a network that actually solves the task.

Enhanced activity spreads from the cue across the object

The most direct evidence for grouping comes from the time course. In an example image with two people, activity increased near the cue and spread across the cued person's representation while staying low outside it, and pixels were progressively labeled as belonging to that person. The same spread occurred through all hGRU layers, at progressively coarser resolutions in higher layers.

To quantify this, the authors compared, at each timestep, the mean activity of lowest-layer units whose receptive fields fell on an object when it was cued versus when another object was cued. That difference increased monotonically over time, indicating progressive enhancement of the cued object relative to distractors. The training objective specified only final semantic labels, not their temporal evolution, so the incremental propagation emerged from learning to segment natural images rather than being explicitly supervised.

Local boundaries first, semantic feedback later

The paper's mechanistic account has two successive stages. Early propagation follows local feature and boundary evidence, so internal edges can temporarily halt the grouping signal and split regions that belong to the same object. Later, category-related activity in higher hGRU layers enables propagation across those internal edges. The network thus combines local boundary evidence with high-level semantic information to build a coherent object representation.

This is the paper's explanation of why earlier boundary-based models fail on natural images: they treat all contours as object boundaries and have no route across internal edges. The authors connect the two stages to cortical processing, where early responses to abrupt feature changes partition a scene into proto-objects and later recurrent processing strengthens true object boundaries over non-object borders. That mapping is the authors' interpretation of how the model relates to visual cortex, not a direct neural measurement in this study.

Reaction times predicted without reaction-time training

The decisive comparison uses human reaction times from a same-different task on the same COCO images: a cue marked an object and participants reported whether a second cue fell on the same or a different object. The four-layer GammaNet explained 19.7% of the variance in those reaction times (95% CI 12–27%), against a noise ceiling of 19.8% (18.2–21.4%). Crucially, GammaNet was never trained on reaction-time data; it was trained only to segment images, so the correspondence emerged from learning the grouping task.

The table separates models that answer different questions. The hand-engineered Growth-Cone model, which operates on object boundaries, explained 16.8% (9.4–24%), and its neural implementation by Mollard et al. explained 15.7% (8.5–23%); both were evaluated on contour-only versions of the images. A recurrent network trained on object boundaries explained 6.1% (1.1–11.1%), and an affinity analysis in pretrained networks explained 7.3% (2–12%). A model trained directly to fit human reaction times reached 20.5% (13–28%), statistically comparable to GammaNet (p=0.82). Layer count mattered: a three-layer GammaNet explained only 4.2% (0–8.3%).

Where the account stops and what it predicts

Two limitations qualify the result. GammaNet receives its input from a separately pretrained and frozen FPN, so the study does not show that the input stage and the recurrent grouping dynamics can be learned jointly end to end. And the model does not explicitly represent border-ownership signals, which indicate which side of a contour belongs to an object; it assigns boundaries to the selected object only implicitly.

The framework also makes predictions that go beyond the current simulations. An ambiguous internal edge should delay grouping when it is close to the cue, because grouping may reach a nearby edge before semantic feedback is available in lower visual areas, but produce little or no delay when it is farther away. Category information about the cued object should become decodable from early visual cortex, including V1, first near the cue and then spreading across the object's retinotopic representation. These are testable predictions for neurophysiological and psychophysical experiments, not results reported here.

The grouping dynamics, not the segmentation score

The paper's most consequential claim is a comparison of training regimes rather than of benchmark accuracy. A model trained only to segment natural images, with no exposure to human reaction times, matched the explanatory power of a model trained directly on those reaction times (19.7% versus 20.5%, p=0.82), while models restricted to contour-only stimuli or to boundary-trained recurrence explained substantially less. The distinguishing feature is that GammaNet propagates a grouping signal for one object at a time through its own recurrent dynamics, as biological grouping of spatially extended stimuli is thought to do.

That is where the result should be read: the value lies in the time-resolved spread of enhanced activity and its agreement with human grouping times, not in beating dedicated segmentation systems. The four-layer depth is part of that claim, since dropping to three layers cut explained variance to 4.2%. Whether the two-stage account holds in cortex remains to be tested by the predictions the authors specify.

Sources used in this answer