Training in Imagination: The Geometry and Economics of Latent World Models

TL;DR

When we train an agent "inside its own head" (imagination), every flaw in its internal physics and reward logic compounds into a massive performance gap. This paper, a collaboration involving Yann LeCun, provides the first rigorous mathematical framework to answer: How much of that gap is the physics engine's fault versus the reward model? And if you have $10,000 to spend on data, how much should you spend on "physics" vs. "labels"?

The Core Tension: Physics vs. Values

In the paradigm of Training in Imagination (exemplified by the Dreamer series), a policy never sees the real world during its update step. It lives entirely within a learned world model.

The Achilles' heel of this approach is Error Compounding. Small errors in the dynamics model ($f$) cause the agent's imagined state to drift away from reality, and errors in the reward model ($r$) give it the wrong objective. Prior theoretical work bundled these together. This paper uncouples them, revealing a fundamental trade-off governed by the "stiffness" (Lipschitz continuity) of the learned representations.

Methodology: The Geometry of Imagination

The authors provide a "Simulation Error Decomposition" that describes the return gap ($|J-\hat{J}|$) as a weighted sum of two terms:

1. Reward Error (${\epsilon }_{rew}$): Scaled by a standard factor ($1/(1-\gamma )$).
2. Dynamics Error (${\epsilon }_{dyn}$): Scaled by a complex coefficient involving the Lipschitz constants of the dynamics, reward, and policy.

1. The Lipschitz Desideratum

A key insight is that low Lipschitz constants are a representation superpower. If you can make your latent representations "smooth" (low $L_f,L_r,L_{\pi }$), you effectively decrease the weight of the dynamics error. This provides a theoretical justification for a recent trend in AI: Temporal Straightening. By forcing latent trajectories to be near-linear, we minimize the "curvature loss," making the imagination more robust.

2. The Scaling Laws of Data

The paper posits that both world models and reward models obey neural scaling laws (power laws). Surprisingly, their experiments show that reward models learn much faster (an exponent of ~0.96 vs ~0.11 for dynamics).

Figure 1: Reward error decays nearly an order of magnitude faster per decade of data than dynamics error.

Optimal Sample Allocation

Theorem 1 is the "budgeting" highlight. It tells a practitioner exactly how to split their budget.

• If dynamics is expensive to learn (small $\alpha$), buy more transitions.
• If the planning horizon is long (high $\gamma$), dynamics error compounds more, so you must favor dynamics data to keep ${\epsilon }_{dyn}$ low.

The "Cheap Trash" Paradox: Fidelity vs. Quantity

Perhaps the most practical section deals with Noisy Rewards. In Reinforcement Learning from Human Feedback (RLHF), "clean" labels are expensive. Is it better to have 100 perfect labels or 1,000 noisy ones?

The authors prove that REINFORCE is surprisingly robust to zero-mean noise. As long as the noise is unbiased, it only increases the variance, which can be beaten down by simply having more trajectories ($1/K$ factor).

• The Takeaway: If your labeling noise has a "floor" that money can't remove, or if doubling your budget doesn't halve the noise (Power law $p<1$), the math says buy the cheapest possible labels and prioritize quantity.

Experiments & Validation

The authors tested these bounds on synthetic benchmarks and LQG (Linear-Quadratic-Gaussian) systems. While the bounds are "loose" (often overshooting the real error by ~30x–200x), they are directionally correct. The proportionality predicted by the math holds up across different value function shapes.

Figure 2: The predicted sample ratio (using the formula) vs. the realized optimal ratio on LQG configurations.

Critical Analysis & Perspective

This work turns "Training in Imagination" from an empirical hack into an engineering discipline.

• Limitation: The analysis assumes a deterministic world model. Extending this to stochastic world models (like Dreamer's RSSM) would require handling the variance of the transition distribution itself.
• The Bias Trap: While the paper shows we can tolerate noise, it warns that bias is fatal. Averaging trajectories can't fix a reward model that is systematically "wrong" (Proposition 2).

For the AI industry, this paper is a signal to stop worrying about "perfect" human feedback and start focusing on unbiased human feedback—and using the surplus budget to make the internal physics of our models as smooth as possible.