GSI-Bench: Can Generative Models Truly Master Spatial Intelligence?

TL;DR

Spatial intelligence is more than just "seeing" 3D—it's about "acting" and "manipulating" it. While most AI benchmarks focus on spatial Q&A, GSI-Bench shifts the focus to Generative Spatial Intelligence (GSI). By challenging models to edit images based on 3D instructions (e.g., "move the apple 15cm left"), researchers found that teaching a model to generate spatial changes actually makes it understand space much better.

The Missing Dimension: Why "Understanding" Isn't Enough

Most current Multimodal Large Language Models (MLLMs) are tested on their ability to describe a scene—identifying if a cat is "on" a mat. However, this is a passive form of intelligence. In the real world, an agent must predict how a scene changes when an object moves.

The authors argue that Generative Spatial Intelligence—the ability to respect and manipulate spatial constraints during image generation—is the "bridge" to true world models. The primary hurdle has been data: real-world images lack the 3D ground truth needed to measure if an edit is "geometrically correct."

Methodology: From 3D Priors to Generative Labels

The core innovation lies in the construction of GSI-Syn and GSI-Real.

1. GSI-Syn (Synthetic Power): Using simulators like AI2-THOR, the team generated 10,500 triplets of (Original Image, Instruction, Edited Image). Because it's a simulator, they have perfect 3D coordinates, allowing for "Spatial Accuracy" metrics that measure millimeters of error.
2. GSI-Real (Real-world Fidelity): Leveraging ScanNet++, they used 3D grounding models to reconstruct the hidden 3D structure of real rooms. This allows them to project 3D bounding boxes onto 2D images to verify if a "removal" or "move" instruction was followed correctly.

Figure: The pipeline for building synthetic and real-world benchmarks through unified scene processing.

The Evaluation Protocol

The authors move beyond "aesthetic" scores, using four objective metrics:

• Instruction Compliance (IC): Did the object move in the right direction?
• Spatial Accuracy (SA): How close is the final 3D pose to the target?
• Edit Locality (EL): Did the model accidentally change the background or other objects?
• Appearance Consistency (AC): Does the object still look like the same object after being moved or scaled?

Experimental Insights: Generation Boosts Reasoning

The researchers fine-tuned BAGEL on the synthetic GSI-Syn dataset. The results were startling:

• Sim-to-Real Transfer: Even though the model only trained on "fake" simulator data, its performance on real-world spatial editing (GSI-Real) jumped by nearly 8 points.
• The "Reverse" Benefit: Remarkably, training on generation improved the model's performance on understanding benchmarks like OmniSpatial. Identifying spatial relationships became easier for the model after it learned how to create them.

Table: Comparison of various models across GSI-Bench. Note the significant boost for BAGEL after GSI-Syn training.

Depth and Visual Evidence

As shown in the qualitative results, many top-tier models (including closed-source ones) struggle with "Edit Locality"—they often warp the background or change the color of the target object when moving it. The fine-tuned BAGEL+ model shows much higher adherence to the original scene's geometry.

Figure: Comparison of spatial editing results. BAGEL+ demonstrates stronger spatial fidelity and better preservation of unaffected content.

Conclusion & Future Outlook

GSI-Bench proves that for AI to move from "chatbots" to "embodied agents," we must prioritize generative spatial reasoning. The fact that generative training improves downstream understanding suggests that LLMs can learn "3D common sense" simply by trying to predict the visual consequences of spatial actions.

Limitations: While the sim-to-real transfer is impressive, global spatial reasoning in highly complex scenes remains a challenge for all current open-source models. Future work will likely need to integrate more native 3D inductive biases into the Transformer architecture itself.