Do vision-language models reason faithfully on spatial tasks or use shortcuts?

Accuracy alone cannot separate reasoning from shortcuts

A model can answer a spatial question correctly without performing spatial inference. If it has memorized common object co-occurrences, such as a cup on a table or a person on a chair, it may produce the right final answer while relying on a linguistic prior rather than the image relations. Standard spatial benchmarks score only the final answer, so they cannot distinguish faithful spatial reasoning from this kind of shortcut.

SpatialChain was built to make that distinction measurable. It attaches an explicit, scene-graph-grounded reasoning chain to each spatially oriented question drawn from GQA, and it evaluates the chain independently of whether the final answer is correct. The paper reports that this protocol surfaces findings invisible to standard accuracy, which is the central claim the rest of the results test.

Chains are synthesized from symbolic scene graphs

Each reasoning chain is synthesized from GQA's structured scene graphs using Claude Haiku 4.5 with extended thinking. A generated chain is retained only when its generated answer matches the ground-truth answer, and it is stored in a format compatible with thinking-mode VLMs. Because each reasoning step derives from a verifiable symbolic annotation, the reference chain can be used to audit whether a model's thinking trace engaged with the correct spatial relations.

The dataset contains 28,350 training examples and 899 test examples, drawn from 19,263 and 719 unique images respectively. Training chains have a mean length of 115 words (median 112), while test chains are slightly longer at a mean of 145 words, reflecting the complexity-biased test selection. The relation distribution is dominated by projective relations: left_of and right_of account for 62.7% of training questions under the first-occurrence labelling scheme, where each question is tagged by the first spatial relation appearing in its string. The paper notes these statistics indicate the entry point of the spatial chain rather than its complete relational content, since many questions combine multiple relations, especially in the test split.

Two axes score overlap and faithfulness separately

The evaluation combines objective chain-overlap metrics with a scene-graph-aware LLM judge. The judge scores reasoning faithfulness and completeness independently of the final answer, so a correct answer can still be flagged when its reasoning is unfaithful. This is what produces the shortcut rate: the fraction of correct test answers whose reasoning the judge marks as unfaithful.

The judge was validated in two ways. Against 198 human-annotated items, judge-human agreement matched human-human agreement. Against a second judge from a different provider, the model ranking was preserved with a correlation of ρ = 0.88. The paper also reports a sweep of the faithfulness threshold, and the shortcut-rate ordering is stable across the range where the metric separates models at all (τ ≥ 0.5).

Four high-accuracy models show shortcut rates above 39%

Applied to nine thinking-enabled VLMs, the protocol found that four of nine models achieved at least 79% VQA accuracy while exhibiting shortcut rates above 39%. In other words, a large share of their correct answers came with reasoning the judge marked as unfaithful. This is the result that standard accuracy cannot expose, because those same answers would count as successes on a conventional benchmark.

The shortcut rate depends on the faithfulness threshold τ. At τ = 0.5, the fine-tuned variants have the lowest rates (Qwen3-VL-8B FT at 0.222, Qwen3-VL-4B FT at 0.194), while InternVL3.5-4B and InternVL3.5-8B have the highest (0.580 and 0.473). The paper reports that this ordering holds throughout the range where the metric separates models at all. The threshold sweep matters because the shortcut rate is not a single fixed number; it is defined relative to a chosen faithfulness cutoff.

Chain quality predicts correctness for seven of nine models

Chain quality significantly predicted answer correctness for seven of the nine models. The two exceptions, Claude Sonnet 4.6 and InternVL3.5-8B, reveal qualitatively distinct failure modes that benchmark accuracy alone conflates: terse output versus verbose-decorative reasoning.

The extended metrics make the distinction concrete. Proprietary models such as Claude 4.6, GPT-5.4, and InternVL3.5-4B show high token precision but very low recall, meaning their compact outputs contain relevant words but cover only a small fraction of the reference content. Open-weight base models such as Qwen3-VL-4B and Qwen3-VL-8B show the reverse, with moderate precision but high recall, consistent with verbose, exploratory chains. Step markers separate the groups further: fine-tuned models average 0.26 to 0.45 per chain, matching the reference format; open-weight base models average 2 to 3, enumerating more steps than the reference; and proprietary models score 0.00, indicating near-total absence of explicit reasoning structure. Entity overlap confirms that fine-tuned models, at roughly 0.48, reference scene-graph objects more faithfully than any other group despite generating shorter chains.

Projective relations dominate and expose grounding gaps

Per-relation accuracy on the test split shows that the aggregate score is dominated by the two well-sampled projective relations: left_of (n = 314) and right_of (n = 327). For those two categories, the ranking of models tracks overall accuracy closely. Several other relations have very small sample sizes due to the complexity-biased test selection, including behind (3 examples), below (4), above (12), and next_to (8), so results for those cells are reported for completeness only and should be interpreted with caution.

The most notable departure is Claude 4.6, which achieves substantially higher accuracy on non-projective relations (in_front_of 0.93, behind 1.00, next_to 1.00) than on projective ones (left_of 0.76, right_of 0.83). The paper interprets this pattern as consistent with Claude 4.6's high shortcut rate: projective left/right distinctions require precise visual grounding that shortcuts cannot supply, whereas relations such as in_front_of are more recoverable from linguistic co-occurrence priors. This is the paper's explanation, not a separately measured causal result.

Fine-tuning cuts shortcut rate but shifts style off-domain

Supervised fine-tuning on SpatialChain improved Qwen3-VL-8B by +6.2 percentage points in-domain and reduced its shortcut rate to 22%, the lowest of all evaluated models. That is the clearest evidence in the paper that training on scene-graph-grounded chains can change both accuracy and faithfulness together.

The same training also produced a stylistic specialization effect on external benchmarks, which the paper proposes to mitigate with replay-augmented training. On ERQA, accuracy for Qwen3-VL-4B dropped from 0.453 at base to 0.388 at step 63, then stabilized between 0.372 and 0.425 for the rest of training. The sharpest per-category drop was in Trajectory Reasoning, from 0.409 to 0.273 at final, a category requiring object-path tracking across multiple images that SpatialChain does not train on. On EmbSpatial-Bench, projective relations (left, right) and above were remarkably stable, with drops under 2 percentage points at final, while depth-related relations bore the bulk of the degradation (close −5.0 pp, far −7.4 pp). The paper reads this as the model applying the SpatialChain template inappropriately to EmbSpatial prompts rather than catastrophically forgetting depth reasoning, and notes a partial recovery for close from step 1,875 onward.

Symbolic relations leave geometry and step correctness untested

The benchmark has boundaries that qualify how far its faithfulness signal transfers. GQA scene graphs were annotated by humans and may contain errors that propagate into the synthetic reasoning chains. The answer-consistency filter verifies a chain's conclusion, not each intermediate step, so the generator can hallucinate: retained chains conclude correctly but are not guaranteed step-correct, and per-step verification against scene-graph node IDs is named as the natural next step.

The faithfulness judge is a single model from the same family as the reference generator, and one evaluated system shares that family. The second-judge and human-validation results bound this risk, but references generated by a non-Claude model remain untested. The task is binary YES/NO with a 58% yes-rate in training, which enables the reported delta but caps the accuracy axis and is amplified by SFT. Finally, scene-graph relations are symbolic rather than metric, so the benchmark does not capture full geometric understanding such as exact distances or 3D depth, and the complexity-biased test split is dominated by projective left/right relations.

The most decision-relevant reading is therefore a comparison, not a single score. When comparing spatial VLMs, the paper's evidence says to look at the pair of numbers: accuracy alongside the shortcut rate at a stated faithfulness threshold, because four of nine models combined at least 79% accuracy with shortcut rates above 39%. The fine-tuned Qwen3-VL-8B at a 22% shortcut rate is the paper's clearest example of improving both axes at once, and the ERQA and EmbSpatial-Bench trajectories show the cost to watch when that improvement is bought through format specialization.

Sources used in this answer