Can success-guided sampling improve mega-scale reinforcement learning for robot control?

Scaling environments wastes experience under uniform sampling

The paper starts from a practical bottleneck in sim-to-real reinforcement learning. Diverse simulator resets let a policy begin episodes from useful intermediate states instead of having to discover everything from scratch, and massively parallel simulation lets many of those episodes run at once. But when task configurations are sampled uniformly, a growing share of each batch lands on configurations the policy has already mastered or cannot yet attempt. Those episodes produce little useful learning signal, so adding parallel environments yields diminishing returns rather than the expected speedup.

SGS targets that allocation problem. Each task configuration is a triple of initial state, goal, and environment configuration: for locomotion, the terrain type and geometry plus start and goal poses; for manipulation, the assembly task, initial robot and object states, and receptacle pose. Before training, the method fixes a set of configurations (32,768 for manipulation, 104,000 for locomotion) and, at each episode reset, chooses one from that set according to how well the policy currently performs on it.

Moderate success rates receive most of the sampling mass

For each configuration, SGS keeps the last 100 Boolean episode outcomes and uses their mean as an online success-rate estimate. It then scores configurations with a Beta-shaped kernel parameterized by a target success rate and a concentration value, and samples the next configuration from a softmax over those scores with temperature 2. The practical effect is that configurations near the target success rate are sampled most often, while very easy and very hard configurations keep a small but nonzero probability.

That nonzero floor matters. A small epsilon term prevents singularities when a success rate is exactly 0 or 1 and keeps every configuration sampleable, so the estimates can be refreshed even for configurations the robot cannot yet solve. The target success rate is adjustable: the paper uses 0.66 for locomotion and 0.5 for manipulation. The authors are explicit that the curriculum principle is not new; Sampling for Learnability already prioritizes levels by p(1−p). SGS's contribution is a simple rolling, Beta-shaped variant combined with diverse resets and very large-scale training.

One reward per domain replaces per-task reward engineering

SGS is paired with two deliberate training choices. First, a single reward function covers all locomotion tasks and another covers all manipulation tasks, each combining a Boolean end-of-episode task-success reward with relatively small per-step robot-specific regularization terms. This removes the need to design a separate shaped reward for each terrain or assembly task. Second, the number of parallel environments is scaled aggressively, which the authors argue only pays off when sampling concentrates rollouts where the policy is actively learning.

The experiments test that argument directly. The scaling study compares SGS against uniform sampling, a hand-designed linear terrain curriculum, and Prioritized Level Replay, which scores configurations by a learnability measure plus a staleness bonus. Evaluation resets are freshly sampled from the training distribution rather than reused from training, and results are means over three training seeds per scale. The number of PPO mini-batches is held constant, so batch size grows with environment count.

Locomotion gains grow with the number of parallel environments

On multi-terrain quadruped locomotion with the Anymal-D robot, a single policy is trained from scratch across 13 terrain types, including slopes, gaps, stairs, balance beams, stepping stones, mazes, and floating islands linked by narrow stones or jump boxes. At one million parallel environments, SGS reaches 73% mean success versus 54% for PLR. The paper reports monotonic improvement with scale for SGS, while the compared methods struggle to benefit or collapse.

The advantage is not confined to the largest scale. At 4,096 parallel environments, SGS achieves nonzero success on many terrains while uniform sampling and the other baselines fail to learn at all. Against Sampling for Learnability, SGS also leads at small scale: 0.456 versus 0.289 best SFL success at 4K environments, and 0.494 versus 0.421 at 32K. The paper reports SFL's final performance alongside its best to show a late-training decline, which is why the comparison uses both numbers rather than a single endpoint.

Assembly tasks separate the samplers most sharply

The manipulation results are where the sampling choice matters most. On Franka nut-and-bolt assembly, where the robot threads an M16 nut onto an upright bolt through roughly 6.5 revolutions, SGS reaches 70% success at one million parallel environments, against 6% for uniform sampling and 5% for PLR. The authors note that uniform sampling had been shown to suffice for easier manipulation tasks, but it does not carry over to this more precise, contact-rich problem.

Scale interacts with task difficulty in a way that changes the comparison. On the easier UR5e rod-in-hole insertion, SGS succeeds at 32K environments where uniform sampling and PLR achieve zero success, but by 256K environments all methods reach about 98%. The sampler's value is therefore largest where the task remains hard at the scale being run, not as a universal constant advantage. The paper also reports that a gravity curriculum used in the main manipulation experiments was later found to be unnecessary.

Distilled RGB policies transfer zero-shot to real hardware

For deployment, the authors train additional UR5e assembly policies with SGS, distill the state-based teachers into RGB policies with DAgger, and transfer them zero-shot to UR5e hardware. Distillation takes about two days on four L40S GPUs with 480 environments each. In simulation, teacher and RGB-student success are close: 90.04% and 89.05% for nut-and-bolt, 95.70% and 93.36% for rod-in-hole, and 96.09% and 96.42% for gear mesh insertion.

Real-hardware success is lower and varies by task: 37.5% (18 of 48 trials) for nut-and-bolt, 61% (30 of 49) for rod-in-hole, and 94% (47 of 50) for gear mesh insertion. First-try success is lower still, at 12.5%, 28.57%, and 76.0% respectively, and throughput ranges from 0.23 to 3.08 successes per minute including resets. The gap between simulated and hardware success is a stated limitation, and closing it is left to future work. The paper does show behaviors such as flipping the nut and pushing the gear against the shaft to reorient it, which the authors describe as robust retrying and dexterous non-prehensile manipulation.

Where SGS stops being the right tool

SGS maintains success-rate estimates over a discrete set of configurations. The paper states that 32,768 cells suffice here, but continuous or much larger task spaces would require a learned or hierarchical scoring representation. SGS also samples from a given distribution rather than constructing it: broad coverage still depends on reset-diversity methods such as OmniReset, and deciding which configurations to make available remains open.

The most decision-relevant reading is a threshold rather than a blanket endorsement. SGS pays off when a task remains hard at the scale being trained and when uniform sampling would spend much of each batch on configurations already mastered or still impossible. On the easier rod-in-hole insertion, that condition fades by 256K environments, where all methods converge near 98%. The paper's own priority is therefore the allocation of experience across configurations, not the raw environment count: scale amplifies SGS's advantage on hard locomotion and nut-and-bolt assembly, but does not by itself rescue uniform sampling.

Sources used in this answer