[OSDI 2025] TensorHub: Breaking the Efficiency-Flexibility Tradeoff in LLM RL Training

TL;DR

Reinforcement Learning for LLMs (like DeepSeek-R1 or O1-style reasoning models) is bottlenecked by the constant need to move TBs of weights between "Trainers" and "Rollout" workers. TensorHub solves this by treating the entire cluster's GPU memory as a virtual, zero-copy storage layer called Reference-Oriented Storage (ROS). It eliminates the 6.7x stall times of traditional methods while supporting elastic, spot-instance, and cross-datacenter workers.

Problem & Motivation: The Weight Transfer Tax

In the RL loop, trainers update model weights, and thousands of rollout workers must immediately "pull" these weights to generate new data. This creates three critical pain points:

1. NCCL Rigidity: While fast, NCCL requires static groups. If one spot instance fails, the whole training job crashes.
2. Storage Bloat: Traditional storage (Ray Plasma, PS) requires "Pushing" weights to CPU/Memory then "Pulling" them back, doubling the traffic and wasting TBs of RAM.
3. Coordination Overhead: Global barriers ensure everyone has the same weight version but force fast workers to wait for stragglers, killing throughput.

The insight of the ByteDance and UW-Madison researchers is simple: The weights are already in GPU memory for inference. Why move them to a 'storage' server at all?

Methodology: Reference-Oriented Storage (ROS)

TensorHub introduces Reference-Oriented Storage (ROS). Instead of a storage server "owning" the bits, a central Reference Server only keeps track of "who has what version in their GPU."

1. The Mutability Contract

To prevent a trainer from overwriting a buffer while a rollout worker is still reading it, ROS enforces a "contract." A worker must publish a version (committing to immutability) and unpublish it before reuse.

2. Pipeline Replication

To prevent the trainer's network link from becoming a bottleneck, TensorHub uses Pipeline Replication. As soon as Rollout-A receives a few chunks of the weight from the Trainer, it can immediately start serving those chunks to Rollout-B.

Figure: Pipeline replication turns bandwidth into a DAG, amplifying throughput as more workers join.

3. Consistency for Model Parallelism

When a model is sharded across 8 GPUs, all 8 must see the same "Latest" version simultaneously to avoid logic divergence. TensorHub solves this with Transactional Semantics: the first request from a group locks in a version for the entire group.

Experiments & Results

The system was tested on massive production workloads at ByteDance, including a 1-Trillion parameter mocked model using 1024 GPUs.

• Bandwidth: TensorHub achieves ~22 GB/s per shard, nearly saturating the 400Gbps RDMA links.
• Standalone Rollouts: Reduced total GPU stall time by 6.7x over NCCL because trainers never have to wait for rollouts to finish pulling.
• Cross-Datacenter: By utilizing "Seeding Replicas" (transferring once over TCP then broadcasting locally via RDMA), it cut stall time by 19x.

Figure: Comparative analysis showing TensorHub's dominance in cross-datacenter and elastic scenarios.

Critical Analysis & Conclusion

Takeaway

TensorHub's ROS abstraction is a masterclass in "Software-Defined Storage" for AI. By moving the complexity to a lightweight metadata server and keeping the heavy data on the "Fast Path" (GPU-to-GPU RDMA), it enables the elasticity required for modern, cost-effective LLM training.

Limitations

• Soft State Reliance: While the reference server is lightweight, its failure during peak burst could briefly stall the discovery of new versions.
• Network Hardware: Its maximum benefits are tied to RDMA-capable (InfiniBand/RoCE) environments.

Future Outlook

As RL training moves toward even more complex flows (like "Best-of-N" speculation and asynchronous off-policy updates), the ability to treat distributed GPU memory as a searchable, versioned object store will become the industry standard.