Do vision-language models match human gaze in cross-modal associations?

Choice agreement hides a spatial gap

Cross-modal associations are systematic pairings between features in different senses, such as the bouba–kiki effect, where people pair the pseudo-word "bouba" with a rounded shape and "kiki" with a spiky one. This study asked whether vision-language models (VLMs) reproduce those pairings and, more unusually, whether they look where humans look while deciding. Humans and models saw the same stimuli: one pseudo-word followed by a pair of images, one rounder and one sharper.

The answer splits in two. Choice alignment, meaning the model selects the same image as humans on matched trials, appeared mainly in a subset of larger VLMs. Spatial alignment, meaning the model's saliency map correlates with the same participant's gaze on the same trial, did not follow: those saliency maps matched human gaze less closely than a center-bias baseline, a fixed Gaussian placed at the center of each image. Choosing like a human and attending like a human are separate results in this study.

A shared trial design with 53 participants

The human reference came from 53 participants contributing 8,208 cleaned trials: 7,141 constructed pseudo-words, 910 adjectives, and 157 canonical trials. Participants read a label aloud and then said A or B to choose the image they thought better matched it, while eye movements were recorded. Trials were kept only if the image presentation was complete, the choice was valid, at least two fixations fell within the two images, and gaze covered at least 80% of image display time.

The label set mixed 4 canonical pseudo-words, 162 phonemically constructed pseudo-words, and 20 adjectives, paired with 75 minimal line-drawing pairs, 92 language-model-generated pairs, and 17 pairs drawn from prior work. Because the same stimuli were used for humans and models, the comparison does not inherit the mismatch of earlier work that evaluated humans and VLMs on different stimuli or tasks.

Human behavior itself was graded rather than binary. Intended-category agreement, the share of responses matching the label's intended round or sharp category, was 94.84% for English adjectives and 84.71% for canonical pseudo-words, replicating the standard effect, but only 73.98% for constructed pseudo-words. Roughly one response in four departed from the intended category, which is why the paper compares models both against the intended category and against the actual response of the participant on that trial.

Gaze accumulates on the chosen image

Gaze is only a useful reference if it carries information about the choice. In the cleaned data, the image a participant looked at more, measured either by exported gaze samples or by summed fixation duration, was the image they chose on 77–80% of trials; for constructed pseudo-words by gaze samples the figure was 80.58%. The image containing the first fixation agreed with the choice at chance, 51.07%.

The authors describe this as an association, not a causal relation: gaze accumulates on the option that is eventually chosen, so it reflects where attention settles rather than an initial side bias. That distinction matters for how the later training experiments should be read, because it is the premise for treating per-trial gaze as a supervision signal rather than as an independent measure of preference.

Larger models match choices, not fixations

Choice alignment was concentrated in larger models. The paper reports that smaller models match adjectives but sit near chance and are side-locked on pseudo-words, while a few larger VLMs align with human choices. Molmo2-8B illustrates how far apart the two abilities can be: near-perfect adjective intended-category agreement of 0.94 alongside near-chance pseudo-word agreement of 0.53.

Spatial results point the other way. For the six generative models evaluated on the 7,141 pseudo-word trials, mean within-image attention–gaze correlation ranged from −0.074 for Molmo2-4B to 0.340 for Gemma4-E4B, while the center baseline reached 0.637 or 0.601 on the same trials. The discussion gives a concrete pairing: Gemma4-26B's pseudo-word choices correlate with human judgments at about r ≈ .58, while its attention correlates with gaze at only .27.

The authors interpret this pattern as models reaching human-aligned choices through holistic image representations rather than the local shape-informative features on which human fixations concentrate. That is the paper's explanation, not a directly measured mechanism; the measured fact is the divergence between choice agreement and spatial agreement.

Fine-tuning choices leaves attention behind

Fine-tuning small VLMs on human choices brought their choice alignment up to the level of a human majority-vote reference on unseen words and images. The gain did not transfer spatially: their attention still matched human gaze less closely than the center-bias baseline. The discussion puts the choice gain at roughly 20 percentage points in human-choice agreement while within-image gaze correlation stayed below the baseline.

This is the paper's sharpest comparison because it holds the model fixed and changes only the training target. If choice agreement were a proxy for human-aligned processing, improving it should have moved spatial alignment toward human gaze. It did not, which is why the authors treat spatial alignment as the more diagnostic test of whether a model processes these associations the way humans do.

Gaze training raises correlation, not choices

Training model attention directly on human gaze raised within-image attention–gaze correlation from below the center-bias baseline to .67–.73, with no detectable change in human-choice agreement or negative log-likelihood. The result looks like successful attention alignment until the control is added: a single average gaze map per image position produced a similar correlation gain.

The authors connect this to prior reports that gaze regularization improves attention alignment, and note that their results agree on attention but that the gain does not carry over to choices and is matched by a constant average map. In this study, then, a higher attention–gaze correlation is not by itself evidence that the model has acquired human-like cross-modal processing, because a map that ignores the word and the trial can produce a comparable increase.

Where the conclusion stops

The fine-tuning evidence rests on three models, one optimization seed, one gaze-loss weight, and the original prompt, so the training results describe those settings rather than a general recipe. Attention maps and Grad-CAM are layer-dependent correlational readouts, not causal explanations of model behavior. Written labels also confound phonology with orthography, and participants were English-speaking members of one institution, shared across splits.

Within those bounds, the paper's contribution is a matched comparison that separates two things usually conflated. Choice alignment and spatial alignment are measured on the same stimuli and the same trials, and they dissociate: larger models can match human choices while their saliency trails a center Gaussian, fine-tuning on choices closes the choice gap without closing the gaze gap, and training on gaze lifts correlation without lifting choices.

The practical priority that follows is about which measurement to trust. When the question is whether a VLM shares human cross-modal associations, agreement on the chosen image is weak evidence on its own, and an attention–gaze correlation is weak evidence unless it beats what a constant average gaze map already achieves. The paper's own threshold is the center-bias baseline: a model's saliency has to do better than a fixed Gaussian at the image center before its spatial behavior can be read as human-aligned rather than incidental.

Sources used in this answer