Physically Grounded 3D Generation: Beyond Vision for Robot In-Hand Perception

In the world of robotic manipulation, the moment an object's geometry matters most—the moment of the grasp—is exactly when vision becomes most unreliable. As a robot's hand wraps around an object, severe occlusion turns 3D reconstruction into a "guessing game" for standard computer vision models.

This paper presents a paradigm shift: Physically Grounded 3D Generative Reconstruction. Instead of asking a model to "hallucinate" what is behind the hand based only on pixels, the authors force the model to respect the laws of physics and the evidence of touch.

The Problem: The "Plausibility" Trap

Current SOTA generative models like SAM3D or Amodal3R produce beautiful 3D meshes. However, in robotic tasks, "looking right" isn't enough. Vision-only models often generate shapes that:

1. Interpenetrate: The object's reconstructed volume occupies the same space as the robot's fingers.
2. Drift in Scale: Without depth or physical anchors, the object's metric size is often guessed incorrectly.
3. Miss Contacts: The surface fails to align with where the robot feels the object.

Methodology: Fusing Vision with Physical Intuition

The authors propose a two-stage approach. The heart of the system is Stage A, which generates the "structure" of the object using a Structure-VAE and a Flow Matching Transformer.

1. The Representation: Pose-Aware SDF

Rather than using point clouds or voxels, the model uses a Signed Distance Field (SDF). This is continuous and differentiable, allowing the system to use gradient-based optimization to "push" the object out of the hand's interior.

2. Multimodal Conditioning

The Flow Transformer doesn't just look at an RGB image. It processes:

• Proprioception: The exact 3D geometry of the robot hand (encoded as a latent).
• Tactile Feedback: Discrete contact points transformed into a volumetric distance tensor.
• Visual Cues: Masked RGB images showing only the visible parts of the object.

Fig 1: The inference pipeline showing the fusion of RGB, hand geometry, and tactile sensors into a shared 3D grid.

3. Physics as a Guide (Training & Inference)

Critically, the authors don't just hope the model learns physics. They enforce it via two specific loss functions:

• Non-Interpenetration Loss: Penalizes the object for overlapping with the hand volume.
• Contact-Consistency Loss: Pulls the object's surface toward the specific voxels where touch was detected.

Experimental Validation

The model was tested using the YCB object set in simulation and then deployed on a real humanoid robot.

Performance under Occlusion

As occlusion increases, the gap between "Vision-Only" and "Physically Grounded" models widens significantly. In the most challenging cases (Bin 5 occlusion), the proposed method maintains high Voxel IoU and Normal Consistency, whereas baselines degrade rapidly.

Fig 2: Qualitative results in simulation. Note how the vision-only baselines (Amodal3R) produce inconsistent shapes that don't respect the hand's presence.

Real-World Transfer

Perhaps the most impressive result is Cross-Embodiment Transfer. The model was trained on Barrett and Allegro hands but successfully reconstructed objects on a real robot using a completely different end-effector and XELA tactile sensors. This proves the shared 3D grid representation is robust enough to generalize across different robot hardware.

Critical Insight: Why This Matters

This work demonstrates that for "Physical AI," generative models must be constrained by the environment. By representing the hand and the object in the same geometric domain, the model implicitly learns Pose Estimation and Amodal Completion simultaneously. The "negative information" (the hand occupies this space, so the object cannot) is just as valuable as the "positive information" of vision.

Conclusion & Future Work

The paper successfully bridges the gap between generative 3D modeling and robotic interaction. While limited by grid resolution (64^3), it provides a blueprint for how future robots might "see" with their hands. Future extensions might include inferring physical properties like friction or mass, further grounding the digital reconstruction in the real physical world.

Paper Metadata:

• Task: Metric-scale amodal object reconstruction.
• Keywords: Physical AI, 3D Reconstruction, Tactile Sensing, Flow Matching.
• Key Results: Voxel IoU 0.586, ADD-S 0.07.