FiDec: Refining Flow Policy via A Local Transport Map
Fisher Decorator: Refining Flow Policy via A Local Transport Map
The Fisher Decorator (FiDec) is a novel flow-based offline reinforcement learning framework that parameterizes policies via a local transport map refined by a Fisher information metric. It addresses the geometric mismatch in existing flow policies, achieving state-of-the-art (SOTA) performance across D4RL, OGBench, and visual manipulation benchmarks.
TL;DR
The Fisher Decorator (FiDec) is a breakthrough in flow-based offline reinforcement learning (RL). By acknowledging that the action space in offline RL is "anisotropic" (having different importance in different directions), FiDec uses the Fisher Information Matrix to guide policy updates. It achieves SOTA performance by ensuring that policy refinements "nudge" actions within the support of the original data rather than drifting into dangerous, unknown regions.
Problem & Motivation: The Geometric Mismatch
Offline RL often boils down to a fundamental trade-off: Expressiveness vs. Stability. We want a policy that can represent complex, multimodal behaviors (Expressiveness), but we must stay near the behavior policy to avoid the "deadly triad" of overestimation and distributional shift (Stability).
Modern state-of-the-art methods like Flow Q-learning (FQL) use flow matching for expressiveness. However, when they refine these policies to maximize reward, they typically use an penalty—an "isotropic" measure.
- The Flaw: An isotropic penalty treats a 0.1-unit change in a high-density action region the same as a 0.1-unit change toward an "out-of-distribution" region.
- The Consequence: In multimodal settings, this causes "mode averaging," where the policy collapses to the mean of two good actions, often resulting in an action that is actually terrible.
Methodology: The Local Transport Map
Instead of re-distilling the entire flow (which is computationally expensive and unstable), FiDec introduces a Local Transport Map.
1. The Residual Formulation
FiDec keeps the base behavior flow and simply learns a "decorator" or residual displacement : This ensures the update is local and "anchored" to the existing data manifold.
2. The Fisher Metric
The core innovation is the Fisher Information Matrix (FIM). The authors prove that the KL-divergence constraint, when applied to a local transport map, is approximately equal to a quadratic form scaled by the FIM: Where is the local Fisher matrix.
Figure: Comparison of standard Flow Q-learning (Left) vs. FiDec's Local Transport Map (Right). Note how FiDec uses the local geometry to "anchor" the refined action.
3. Estimating the Score from Flow
One "magic" property of flow matching is that the Score Function (the gradient of log-density) is embedded in the velocity field :
abla_a \log \pi_\beta(a|s) \approx \frac{t_\epsilon v_\beta(t_\epsilon, s, a) - a}{1 - t_\epsilon}$$ By perturbing the time variable slightly ($t_\epsilon \approx 0.8$), the authors can extract the local geometry of the behavior distribution without extra training. ## Experiments: Breaking the Bottleneck FiDec was tested against 11 baselines including Gaussian, Diffusion, and Flow-based policies. ### Offline Performance On **OGBench**, which features high-dimensional humanoid and puzzle tasks, FiDec showed significant gains. For example, it outperformed DeFlow in 10 out of 11 categories.  *Table: Comparison of FiDec against SOTA Gaussian, Diffusion, and Flow Policies. FiDec excels in multimodal antmaze and humanoid tasks.* ### The Anisotropic Advantage Why does it work? In Figure 3 of the paper, the authors visualize how $L_2$ (isotropic) methods collapse. - **FQL/DeFlow**: Often collapse *between* modes where the reward is low. - **FiDec**: Follows the high-density "ridges" of the behavior distribution to find the best action *within* the valid support.  *Figure: Isotropic vs. Anisotropic refinement. FiDec (Blue) respects the underlying distribution manifold (Red), while isotropic methods drift or average out.* ## Efficiency & Online Fine-Tuning Offline-to-Online fine-tuning is a major challenge in RL. FiDec provides a "Strong Prior." Because it respects the geometry, the transition from offline data to online exploration is smooth. In **Antsoccer** and **Humanoid Maze**, FiDec reached success rates near 100% while methods like IQL struggled to improve beyond their offline starting points. ## Conclusion & Critical Analysis FiDec is a theoretically elegant and practically efficient solution for offline RL. Its beauty lies in revealing that **Information Geometry** is not just for theorists—it can be used to stabilize high-performance generative policies in real-time. **Limitations**: 1. **Score Approximation**: It relies on the flow model being well-trained. If the flow $v_\beta$ is noisy, the Fisher estimate will be poor. 2. **Hyperparameter $t_\epsilon$**: While the paper provides a "first principles" derivation for $t_\epsilon \in [0.7, 0.8]$, it still requires some tuning depending on the dataset's complexity. Overall, the Fisher Decorator proves that when it comes to refining complex policies, **the shortest path (Euclidean $L_2$) isn't always the best path—the best path is the one that respects the data's geometry.**