Can individual world latents be provably recovered without reconstruction?

A good predictor can still hide every variable

Joint-embedding predictive architectures (JEPAs) learn a world model by predicting in representation space rather than reconstructing observations. That choice removes the anchor that other identifiability work uses: no decoder, observation likelihood, or auxiliary signal ties a learned coordinate to a variable of the world. The paper states that avoiding collapse is not the same as recovering latents, because a non-collapsed representation may still encode the true factors as an invertible mixture.

The paper's motivating case is a household robot asked to slide a cup left while keeping a nearby knife fixed. If one learned latent mixes cup position with knife position, changing that latent is no longer a targeted cup movement. The paper therefore separates two questions that a single predictive score conflates: is the latent state present in the representation at all, and has it been resolved into individual variables?

Dense readout and latent MCC measure different things

The paper keeps two metric families apart. The dense readout R²(h→z) is the coefficient of determination of the best linear map from the whole estimated vector onto all ground-truth latents. Any invertible transformation of the estimated latents leaves it unchanged, so a high value says the information is present but says nothing about whether coordinates are separated.

Latent MCC scores the best one-to-one matching of estimated to true coordinates using Pearson correlation, maximized over permutations. It reaches one precisely when the representation recovers individual world latents in the paper's sense. This is why the paper reports MCC as its main individual-latent recovery score, and why a method can preserve dense R² while improving MCC — the two are not substitutes.

Structural Diversity asks which features a factor touches

Structural Diversity is the paper's key condition: different latents leave distinct dependency footprints on observations, described in the abstract as no two snowflakes being alike. DSReg, short for Dependency-Sparsity Regularization, turns this into an objective on the dependency Jacobian — how observed variables respond to a rotation applied to the representation. The rotation is chosen so that this footprint is sparse.

The paper argues the condition is mild in a specific sense: as a condition on dependency footprints, Structural Diversity is strictly weaker than all structural conditions of prior identifiable latent variable models. It also names the boundary the condition cannot cross — factors with identical footprints cannot be distinguished without extra information.

The fix is a rotation, not a retrained encoder

DSReg applies post hoc to any linearly identified representation, reusing trained checkpoints, and the paper reports no loss relative to joint training. Because a rotation loses no information, dense performance is untouched while the coordinates are rearranged into individual latents. The paper reports that the method can be fit jointly with predictive learning or afterwards.

The theory comes in two steps. Under the LeJEPA premise and Structural Diversity, DSReg recovers individual world latents up to signed permutation. A second result, Theorem 3, treats Jacobian estimation error: with a threshold τ and error bound ε satisfying τ+ε below a margin σρ*(δ), every minimizer of the thresholded population criterion lands within δ of a signed permutation. The paper notes exact recovery is impossible under noise, since thresholding at any τ>0 ties the count on rotations close to a signed permutation, so δ-closeness is the appropriate notion.

Gains hold on images the authors did not generate

On Gaussian 3DShapes, an external DeepMind renderer with six factors, the paper reports latent MCC of 0.921±0.005 for DSReg against 0.640±0.041 for LeJEPA, 0.272±0.025 for β-VAE, and 0.277±0.043 for β-TCVAE. Dense R² is shared between LeJEPA and DSReg at 0.868±0.003 and 0.861±0.002 respectively, which is the paper's evidence that the rotation preserves the span. The β-VAE and β-TCVAE rows are external references with a matched trunk, trained on i.i.d. images at an untuned β=4; the paper states their low dense R² reflects a linear readout only, with nonlinear informativeness staying high, so the gap measures axis alignment rather than lost information.

The per-factor split is the more decision-relevant result. The three hue factors paint large pixel regions and every method captures them to some degree, while scale, shape, and orientation move few pixels, and there the VAE baselines collapse outright. The paper's explanation is that reconstruction-driven objectives weight factors by the pixels they explain, whereas the dependency criterion asks only which features a factor touches, so pixel-poor and pixel-rich factors fare alike.

A second renderer, quarter-orientation dSprites, is presented as a stress test under an imperfect linear stage: a symmetry of the sprites breaks the linear premise unless the factor grid is restricted. Over twenty runs, DSReg improves over LeJEPA and exceeds both VAE baselines, with one-sided Mann–Whitney p=8×10⁻⁵ against β-VAE and p=4×10⁻⁵ against β-TCVAE. The paper reads this as evidence that the premise, once attained, is the bottleneck rather than the rotation step.

Latent-only baselines cannot break the symmetry

The paper gives simpler baselines two chances. On the exact analytic orbit, the latents are isotropic Gaussian, so PCA, Varimax, and FastICA face the very symmetry that motivates DSReg and stay at the unrotated level alongside LeJEPA and a random orbit, while DSReg from exact Jacobians approaches the oracle. The residual gap is attributed to the ℓ1 relaxation rather than estimation error: on a fraction of random worlds the relaxation's global minimizer is a slightly mixed rotation even though the support criterion still identifies the permutation.

On learned, non-isotropic representations the baselines have a real opening, yet the paper reports the latents still mixed while DSReg nearly matches the supervised Procrustes oracle. A stronger baseline that consumes the same dependency signal — the scale-invariant column-support sparse-ICA criterion of Ng et al. optimized on the same estimated anchor Jacobians — is also tested. The paper's conclusion is that the observation-side dependency signal, not the span alone, does the work.

Sparse access improves while dense readouts stay put

The paper's third question is whether a downstream module restricted to a few coordinates can do its job. Its probes are visual editing, sparse model predictive control, five-step rollout prediction, and detection of physically implausible next states. A recurring few-shot single-latent readout selects the one estimated coordinate that correlates best with a factor and fits a scalar affine map from that coordinate alone, forbidding any mixing; on a representation whose coordinates are individual factors a few labels suffice, while on a mixture no scalar map works at any label budget.

The paper reports that dense readouts and retrieval are unchanged to two decimals on every task, so what DSReg gains over the baselines is gained in sparse access rather than in the content those readouts measure. It also reports that recovery is insensitive to the sparsity surrogate and the anchor budget, and that long-tail and correlated training distributions leave the recovery gain intact with dense R² between 0.80 and 0.93.

Simulated worlds first, physical robots later

The paper states its clearest limitation directly: benchmarks are simulated or rendered, and DSReg has not yet been tested on a physical robot. The Gaussian latent-world premise is described as a foothold rather than a ceiling, with every extension of linear identifiability to richer worlds widening the guarantee. Identical footprints remain the one boundary that support alone cannot cross, and the paper points to temporal structure or cheap interventions as candidate signals there.

The practical priority that follows is narrow and specific. If a representation already passes the dense test — the latent state is present — then the question worth asking is whether its coordinates can be acted on one at a time, and latent MCC plus a few-shot single-latent readout answer that question where dense R² cannot. If two factors genuinely share a dependency footprint, no rotation from this criterion will separate them, and the paper's own framing says extra information is required.

Sources used in this answer