Can RFPO make flow policies reliable under one-step execution?

The gap sits between training and deployment

Flow-based policies generate actions by numerically integrating a learned velocity field from noise to an action. Training and evaluation typically use a full budget of 64 Euler steps, but deployment may only afford one. The paper calls the resulting mismatch the few-step discretization gap: a policy optimized under full-step execution is not explicitly constrained to stay reliable when the solver is coarse, so cutting the step count can substantially change the generated action and degrade control.

This matters because the problem is not simply that fewer steps add numerical noise. The paper's framing is that the training objective optimizes return at the training integration budget without shaping the vector field for coarse integration. A high-reward flow policy can therefore still be fragile at one step, which is exactly the failure the study sets out to fix.

Two training signals shape the student policy

RFPO builds on FPO++, keeping its original on-policy actor–critic objective, and adds two complementary signals during training. The first is reward-aware online Reflow. For each observation the current student draws four auxiliary initializations, with a 25% chance of a zero start and otherwise Gaussian noise, and produces a full-budget endpoint with the gradient stopped. A straight path between that initialization and the student's own endpoint becomes the regression target, so the transport geometry the coarse solver will encounter is repeatedly straightened. Samples with positive advantage are weighted more heavily, focusing rectification on behavior useful for control.

The second signal is multi-budget action distillation. A frozen Gaussian PPO controller supplies action targets, and the same flow student is trained to match those targets at budgets of 64, 8 and 4 Euler steps. Notably, one step is excluded from this set, so one-step behavior is never directly supervised by the teacher. A third, lighter regularizer exposes the student to budgets of 1, 4, 8, 16 and 64 through a learned budget predictor, encouraging consistency with the full-budget action while penalizing expected solver cost.

At deployment all of this is stripped away. The PPO teacher and budget predictor are removed, the observation is paired with zero noise, and the student runs a single Euler step. The deployed controller is therefore just the learned flow policy, and the efficiency gain comes from the solver budget rather than from a smaller network.

One-step returns stay near full-step returns

The cross-embodiment table is the paper's central evidence. Under zero initialization, moving from 64 Euler steps to one changed average episodic return by −1.5% on Go2, +0.4% on Spot, −0.9% on H1 and +1.7% on G1. Under random initialization the 64-to-1-step changes stayed within 2.4% across all four embodiments. The authors read this as evidence that the learned one-step behavior is neither tied to a single robot morphology nor dependent on deterministic zero initialization.

The baselines show how unusual that flatness is. FPO++ degrades severely on every task: on Go2 its zero-initialized return falls from 21.29 at 64 steps to −187.70 at one step, and on Spot from 9.29 to −404.05. Distillation alone does not rescue the problem. KD-only improves full-budget control on some embodiments but still collapses at one step, reaching −4919.70 on Go2 and roughly −1.5×10⁵ on Spot. Because reward scales differ across tasks, the paper cautions that comparisons should be made within each embodiment rather than across rows.

Transport rectification does most of the work

The Go2 ablation separates the two mechanisms. FPO++ drops from 21.29 at 64 steps to −187.70 at one step, and KD-only falls to −4919.70. Uniform Reflow, without the reward weighting, keeps a nearly flat profile from 33.57 at 64 steps to 32.84 at one step, and reward-aware Reflow shows similar numerical robustness. Combining Reflow with action-space supervision raises the full-budget controller further while preserving few-step performance. The authors conclude that transport rectification is the primary mechanism for robust coarse integration, with distillation supplying complementary control supervision.

Reward alone does not guarantee tracking

The paper is careful to distinguish episodic reward from low-level control quality. On Go2, FPO++ and KD-only show large one-step forward-velocity tracking errors of 0.883 and 0.810, with fall fractions of 0.680 and 0.334. Uniform Reflow cuts tracking error to 0.070 with no observed falls.

The instructive case is reward-aware Reflow alone: it keeps a high one-step episodic reward but incurs a tracking error of 0.805. High reward under one-step execution therefore does not by itself prove that the robot follows commands well. The full RFPO combination is what the paper reports as maintaining stable command tracking, which is why the reward table and the tracking table need to be read together rather than as substitutes.

Latency gains measured on the onboard CPU

The efficiency claim is specific to the Go2 onboard setting. Inference was measured with single-thread ONNX Runtime on the Go2 onboard CPU after 30 warm-up iterations and 100 timed evaluations, reporting mean and p99 latency. Under that protocol, one-step execution retained 98.5% of the 64-step reward while mean latency fell from 4.39 ms to 0.08 ms, a 54.9× speedup.

This number should not be read as a general speedup for every embodiment or every deployment stack, because the measurement is tied to one robot's onboard CPU and one runtime configuration. What it does establish is that the solver budget, not just the network, is a meaningful part of the inference cost in this setting.

Where the evidence stops

Simulation covers two quadrupeds and two humanoids, but real-robot validation focuses on Unitree Go2 locomotion with a limited set of motion commands. The matched-command hardware analysis reports joint-position correlation and mean absolute error, measuring joint-position consistency rather than full state-level sim-to-real equivalence. Broader real-world testing on humanoid platforms and more contact-rich tasks remains open.

The efficiency also has a training-side price. Online Reflow needs full-budget student rollouts to build student-induced endpoints, and distillation depends on a separately trained frozen PPO controller. Deployment uses a fixed one-step Euler solver; the adaptive compute regularizer is training-only, and the authors suggest that selecting the integration budget online from the current control state could improve the accuracy–efficiency trade-off for highly dynamic or contact-rich behavior.

What to take from the comparison

The paper's most decision-relevant result is not that one-step execution is fast, but that the gap between 64-step and one-step returns can be closed by shaping the transport path during on-policy reward optimization. The ablation supports a clear priority: Reflow-style rectification is what keeps returns flat as the solver budget shrinks, while action distillation mainly improves the full-budget controller. A reader weighing this approach should therefore check two things separately: whether one-step reward holds up, and whether command tracking and fall fraction hold up, since reward-aware Reflow alone shows that the first can survive while the second does not.

The scope condition that most changes the answer is the deployment regime. RFPO's evidence supports a fixed one-step Euler student on the four tested embodiments, with hardware confirmation limited to Go2 locomotion. For contact-rich or highly dynamic tasks, the paper's own suggestion of online budget selection points to where a fixed one-step policy may need to be revisited.

Sources used in this answer