[ICML 2025] RCRL: Steerable Policies and Robust Learning via Reward Conditioning

TL;DR

Reward-Conditioned Reinforcement Learning (RCRL) is a novel framework that transforms how agents perceive objectives. By conditioning policies on reward parameterizations and learning from "counterfactual" rewards off-policy, RCRL produces agents that are not only more robust in their primary task but are also "steerable" at deployment—allowing users to adjust behaviors like speed or effort without any retraining.

Background: The Brittle Nature of Fixed Rewards

In standard Reinforcement Learning, the reward function is a black box. Once a policy is trained on $r=w_1\cdot extvelocity+w_2\cdot extenergy$, it is stuck. If you later decide you want a more energy-efficient gait, you must restart from scratch. Furthermore, minor reward misspecifications often lead to suboptimal behaviors that are hard to correct. RCRL argues that we should treat the reward function as a condition rather than a constant.

Methodology: High-Efficiency Off-Policy Learning

The core insight of RCRL is that a single trajectory contains useful information for many different reward functions.

1. Reward Parameterization

RCRL assumes honors a structured reward where $r_{\psi }(s,a)=f(\psi ,c_1,\dots ,c_k)$. Here, $\psi$ represents weights or parameters. The agent interacts with the world using a nominal reward ${\psi }^{\star }$, but the replay buffer stores the raw components ($c_1,\dots ,c_k$).

2. Dual-Track Updates

During training, RCRL samples a mixture of:

• Nominal Updates: 50% of the batch uses the target ${\psi }^{\star }$.
• Counterfactual Updates: 50% of the batch uses sampled alternative $\psi$ values.

This ensures the agent learns a manifold of behaviors. If you change the input $\psi$ at test time, the policy responds by shifting its behavior accordingly.

Experimental Validation: Robustness and Steerability

Performance Boost

Even if your only goal is the nominal task, RCRL acts as a powerful regularizer. On the DeepMind Control Suite and HumanoidBench, SIMBAV2 + RCRL consistently outperformed the base algorithm. This suggests that "seeing" other ways to solve similar problems helps the agent learn more robust representations.

Zero-Shot Steerability

Perhaps the most impressive result is the "Steerability" shown in Figure 5. By simply sliding a metaphorical "speed dial" (changing the input $\psi$), a Cheetah agent trained only for general running could precisely target specific velocities without ever being explicitly trained on them during environment interaction.

Critical Analysis & Conclusion

Takeaway

RCRL effectively bridges the gap between single-task and multi-task RL. It offers the simplicity of single-task data collection with the versatility of multi-task policies. It is particularly valuable for complex robots (like Humanoids) where manual reward tuning is a nightmare.

Limitations

The primary limitation is the distribution shift. Since the agent only explores under the nominal policy, it may struggle to learn behaviors that require radically different exploration strategies (e.g., if the nominal task is "walking," it might never discover "jumping" via off-policy rewards alone).

Future Work

The next frontier for RCRL involves automated reward discovery—determining which auxiliary reward parameterizations provide the most synergy to the main task to avoid gradient interference.