[UC San Diego] XL-VLA: Bridging the Morphology Gap in Dexterous Robotic Manipulation

TL;DR

XL-VLA introduces a unified latent action space that allows a single Vision-Language-Action (VLA) model to control multiple, kinematically diverse dexterous hands. By mapping complex joint-space actions into an embodiment-invariant latent manifold, the framework achieves a +40% mean success rate improvement over standard baselines and enables zero-shot transfer to new robot-task pairings.

Background: The Vocabulary Problem in Robotics

In the world of Large Language Models (LLMs), "vocabulary" is universal. In robotics, however, the "vocabulary" of actions is tied to the hardware—the number of joints, their limits, and their arrangement. This is particularly problematic for dexterous manipulation, where hands like the Ability Hand (5 fingers, simple actuation) and the Paxini DexH13 (4 fingers, high DoF) "speak" different languages. Training a VLA model usually means collecting data for every specific hand, which is unscalable.

Why Current VLA Models Fail at Scaling

Previous SOTA models like π0 or RT-2 treat actions as tokens. While effective for simple grippers, they struggle with dexterous hands because:

1. Dimensionality Mismatch: Different hands have different total Degrees of Freedom (DoF).
2. Kinematic Variation: The same joint angle on two different hands might result in completely different fingertip positions.
3. Data Fragmentation: Data collected on one hand is typically useless for another without manual, lossy retargeting.

The Innovation: A Shared Latent Action Repository

The core "Aha!" moment of XL-VLA is treating the hand as an interface rather than the core of the policy. The authors built an unsupervised latent autoencoder that sits between the VLA brain and the robotic fingers.

Architecture Overview

The system consists of a VLM backbone (PaliGemma) and an Action Expert. Instead of predicting raw joint angles, the expert predicts latent tokens.

The Unsupervised Training Trick

To ensure the latent space is truly "embodiment-invariant," the authors used three clever constraints during a pre-training phase that requires zero human demonstrations:

• Reconstruction Loss (L1): Ensures each hand's autoencoder is accurate.
• Retargeting Loss (L2): Uses Differentiable Forward Kinematics (FK). It forces the latent code for a "pinch" on one hand to decode into a geometric "pinch" on another hand by minimizing the difference in fingertip distances.
• KL Regularization (L3): Smooths the latent space into a Gaussian distribution, making it easier for the VLA model to learn and interpolate.

Experimental Results: Scaling Across Bodies

The researchers tested XL-VLA on 10 tasks (e.g., Prepare Fruits, Stack Cans, Hand over Bottle) using 4 diverse hands.

Key Findings:

• Success Rates: XL-VLA achieved a 90% success rate on "Hand over Bottle", whereas the baseline π0 (operating in raw joint space) managed only 40%.
• Zero-Shot Capability: Even when the model never saw a specific hand perform a specific task, it generalized successfully. For example, if it learned to "re-arrange boxes" on an XHand, it could immediately execute it on a Paxini hand.
• Humanoid Integration: When applied to the Unitree G1 humanoid, the latent alignment allowed it to benefit from data collected on stationary xArms, boosting performance by 57%.

Critical Insights & Future Outlook

XL-VLA proves that the bottleneck in cross-embodiment learning isn't necessarily the complexity of the task, but the representation of the action. By abstracting the hardware away into a latent manifold, we can finally treat robotic actions more like words in a language.

Limitations: The current framework depends on having 3D URDF models for the hands to compute the differentiable FK. In environments where URDFs are unknown or inaccurate, the geometric alignment (L2 loss) might suffer.

Overall, this is a major step toward a "Foundation Model for Motion" that isn't trapped inside a single robot's body.