VLA Foundry: Breaking the Silos Between LLMs, VLMs, and Robotic Actions
VLA Foundry: A Unified Framework for Training Vision-Language-Action Models
VLA Foundry is a unified open-source framework from Toyota Research Institute that integrates the training of Large Language Models (LLM), Vision-Language Models (VLM), and Vision-Language-Action (VLA) models. The framework introduces two primary models: Foundry-VLA-1.7B (trained from scratch) and Foundry-Qwen3VLA-2.1B-MT (leveraging a pretrained backbone), with the latter achieving a significant performance lead on the LBM Eval simulator.
TL;DR
Toyota Research Institute (TRI) has released VLA Foundry, an end-to-end open-source framework that bridges the gap between general-purpose AI and embodied robotics. Unlike previous tools that only focus on the final "Action" layer, VLA Foundry allows for total control over the LLM → VLM → VLA pipeline. By releasing two robust models (1.7B from-scratch and 2.1B Qwen-based), TRI demonstrates that a unified training stack can outperform specialized closed-source baselines by over 20%.
Beyond "Just Fine-Tuning": The Motivation
The field of robotics is currently suffering from a data scarcity paradox. While LLMs have trillions of tokens to learn from, robot interaction data is rare and expensive. Most researchers handle this by taking a "pre-baked" VLM (like CLIP or LLaVA) and tacking on a robot action head.
The authors of VLA Foundry argue this is insufficient. Decisions made during initial language pretraining—such as how many tokens represent spatial relationships—directly dictate how well a robot can eventually "understand" an instruction like "put the red bell pepper in the bin." VLA Foundry was built to give researchers the keys to the entire factory, not just the assembly line.
Methodology: The Unified Stack
VLA Foundry’s architecture is built on four pillars: Modularity, Hackability, Performance, and Reproducibility.
1. The Architecture Pipeline
The framework supports a staged growth of a model:
- LLM Phase: Training a standard Transformer on text data (e.g., DCLM dataset).
- VLM Phase: Integrating a Vision Transformer (ViT) with the LLM via "pixel-shuffle" pooling to handle image-caption pairs.
- VLA Phase: Adding a Flow Transformer action head. This head uses features from the VLM's frozen latent layers to denoise action sequences, enabling complex bimanual manipulation.
Figure: The VLA pipeline, showing how visual tokens and observation tokens are fused within the LLM layers to feed the action flow transformer.
2. Robotics-Specific Engineering
The framework solves several "hidden" headaches in robotics research:
- Action Chunking: Predicting windows of future timesteps to ensure smooth, non-jittery motion.
- Unified Normalization: Handling data from different robot stations using percentile-based scaling to mitigate outliers.
- Hybrid Mixing: A probabilistic dataloader that can mix raw text, image-caption data, and robot demonstrations in precise ratios during training.
Experimental Results: Scaling Wins
TRI evaluated their models using LBM Eval, a high-fidelity simulator powered by the Drake physics engine.
Key Performance Insights:
- Backbone Matters: The model built on the pretrained Qwen3-VL backbone (2.1B) decimated the from-scratch version and the previous SOTA (LBM-MT). It achieved an average success rate increase of 23 percentage points.
- Scaling Efficiency: The framework demonstrated linear scaling across up to 16 nodes (128 H100 GPUs), proving it is ready for massive research-grade workloads.
Figure: Comparisons on 16 simulation tasks show that leveraging a strong pretrained VLM (Qwen3VLA) leads to superior robotic success compared to training from scratch or using older CLIP-based architectures.
Impact: A New Standard for Open Robotics
VLA Foundry isn't just a model; it's a research infrastructure. By providing the community with the same tools used at industrial labs like TRI, the authors are lowering the barrier to entry for "Foundation Model" robotics.
Takeaways for the Industry:
- Software-Hardware Co-Design: The framework’s success on bimanual (two-armed) tasks indicates that VLAs are finally becoming capable enough for complex, coordinated household or industrial labor.
- Open Source is Catching Up: The fact that a fully open model can now match or exceed prior closed-source industrial models is a milestone for the ecosystem.
Future Work & Limitations
While powerful, VLA Foundry currently focuses on Flow-Matching heads. While other heads like Diffusion are implemented, more work is needed to explore Autoregressive discrete actions (like RT-2). Additionally, while simulation results are dominant, the "Sim-to-Real" gap remains a challenge that requires further multi-modal data recipe tuning.
VLA Foundry is available on GitHub and Hugging Face, providing a complete kit for the next generation of embodied AI researchers.