Can Large Vision-Language Models Internalize Active Mentalization for Spatial Reasoning?

Active mentalization means planning a view, then imagining it

The paper names its target ability Active Spatial Mentalization (ASM): when a spatial question cannot be settled from the given images, the model decides which viewpoint would supply the missing evidence, imagines the scene from that viewpoint, and then reasons over the imagined observation. The authors treat this as an intrinsic part of visual intelligence that current LVLMs largely lack, rather than as a separate tool they can call.

The distinction matters because the paper positions ASM against two existing patterns. Prior data-centric approaches mostly supply answer-only supervision, so the model learns the final label but not how to acquire missing evidence. Tool-based approaches instead call an external world model during inference, which adds intermediate observations and extra inference time. SpatialASM aims for a third route: the plan-imagine-reason sequence is produced by the LVLM itself, after being learned during post-training.

Training simulates viewpoints the model has not seen

The framework starts from existing spatial question-answer pairs and augments them with synthesized ASM trajectories. An allocentric novel-view generator, trained on scene data, renders what the scene would look like from a hypothetical viewpoint. The word allocentric is doing real work here: the target viewpoint is specified by marking a pixel location on an anchor image plus a viewing direction, rather than by an ego-centric camera motion such as moving forward a fixed distance.

Two forms of supervision are then built on top of these rendered views. Explicit ASM trajectories teach the model when and how to invoke view planning and imagination inside a reasoning chain. Automatically constructed perspective-taking tasks strengthen viewpoint-conditioned imagination in two complementary directions: generated-view localization, which asks the model to recover the allocentric viewpoint that produced an observation, and pose-conditioned view selection, which asks it to pick the candidate image matching a described location and direction. Both answers are derived automatically, without human annotation.

Rendered views are filtered by geometric consistency

A synthesized trajectory is only useful if the imagined view is geometrically plausible. The paper uses a geometric consistency measure, GeoP, to score how well a generated image agrees with the input scene, and discards samples whose candidate views exceed a threshold. For the perspective-taking tasks, the ground-truth candidate must be geometrically consistent with the input for the reward to be meaningful, so candidates are selected with GeoP and samples with no acceptable candidate are dropped.

The filtering threshold is deliberately lenient in one place: because reasoning correctness is also filtered during supervised fine-tuning, the authors set the GeoP threshold to 2.5 when screening candidate views. The ablation in the paper compares training with and without these filtering steps, so the filtering is treated as a design choice to be tested rather than an assumed necessity.

Accuracy rises across seven benchmarks and four backbones

The evaluation spans seven benchmarks, ranging from relatively perception-oriented spatial understanding to harder tasks that require multi-image, cross-view and viewpoint-transformation reasoning. Across these seven benchmarks, SpatialASM improves average accuracy for LVLMs built on four different backbones. The abstract and discussion both describe the improvement as consistent across those backbones, and the reported general-VQA cost is small: average performance on three general-VQA benchmarks stays within 0.6 points of the original models.

That pairing is the paper's central empirical claim. A spatial-training intervention that lifted spatial scores while badly degrading ordinary visual question answering would be a poor trade; here the authors report the spatial gain and the general-capability retention together. The benchmarks are aggregated as an average, so the headline number describes overall spatial accuracy rather than performance on any single task type.

The model learns a habit, not a fixed reasoning template

A natural worry is that training on ASM trajectories simply teaches the model to emit the same plan-imagine-reason script every time. The paper's qualitative comparison argues against that. In most cases the trained model does not explicitly follow the ASM pattern, yet its overall spatial accuracy still improves substantially, which the authors read as evidence that the training internalizes generalizable spatial reasoning rather than imposing a rigid template.

The same comparison shows the model adapting its behavior to task type and difficulty: on some perspective questions it answers correctly without producing an explicit ASM chain, while on harder multi-image tasks it generates active spatial planning and imagination steps that lead to the correct answer. This is the paper's interpretation of qualitative examples, not a separately quantified measure, so it should be read alongside the benchmark averages rather than as an independent result.

Imagined views can be more informative than small camera moves

The paper also compares its novel-view generator against other rendering options, including GPT-Image-2, SpatialEdit and SVC, using median GeoP rather than mean GeoP because a few poor-quality images with values above 10 would otherwise dominate the average. In one case the two generators target roughly the same viewpoint and SpatialASM-NVG achieves the lower GeoP, 0.0599 versus 0.2407, because objects in its output more closely resemble their counterparts in the input images.

A second case complicates any simple reading of GeoP as quality. SVC's output is more geometrically consistent with the inputs, with GeoP 0.1266 against 0.1284, but its ego-centric formulation produces only small camera motions, so the output stays highly similar to the anchor image and adds little information for downstream reasoning. The allocentric formulation instead induces larger viewpoint changes. The paper's point is that geometric consistency alone does not capture whether an imagined view is useful for answering the question.

The evidence covers post-training, not open-world scenes

The training data are synthesized from existing spatial QA sources, and the paper reports that the 66k training subset used for supervised fine-tuning contains no sample synthesized from the Mindcube training set, to avoid distribution overlap with final evaluation. The trajectory synthesis relies on pretrained LVLMs to produce planning and reasoning traces and captions, and on the novel-view generator for the imagined observations, so the quality of the supervision is bounded by those components.

The scope is also stated by the authors themselves: extending ASM to dynamic real-world scenes, embodied interaction and longer-horizon 3D reasoning is listed as future work. The demonstrated result is that a post-training framework can internalize viewpoint planning and imagination into several LVLM backbones and raise average spatial accuracy on seven benchmarks without materially hurting general VQA. Whether the same internalization holds when scenes move, when the model must act physically, or when reasoning spans much longer horizons is not established here.

Supervision design matters more than adding a renderer

The paper's most decision-relevant comparison is not between model sizes but between ways of supplying spatial supervision. Answer-only data gives the model a label without teaching it how to obtain missing evidence; external world-model calls give it evidence but at the cost of extra intermediate observations and inference time; SpatialASM instead synthesizes plan-imagine-reason trajectories so the capability is learned during post-training and produced intrinsically at test time. The reported outcome, average gains on seven spatial benchmarks across four backbones with general-VQA performance within 0.6 points, is the evidence for that choice.

Within that framing, the practical threshold the paper offers is about what to supervise, not how much to render. The trajectory synthesis needs no extra metadata such as bird's-eye-view images or bounding boxes, which is what makes it scalable across existing spatial datasets, and the GeoP filtering plus the lenient 2.5 threshold show where the authors chose to trade synthesis strictness for volume. Readers weighing spatial post-training should therefore focus on whether the supervision teaches viewpoint selection and imagination, and on whether imagined views are informative rather than merely consistent, since the SVC comparison shows those two properties can diverge.

Sources used in this answer