[ICLR 2025] HiLAM: Discovering Hierarchical Skills from Actionless Videos via Dynamic Chunking

Executive Summary

TL;DR: HiLAM introduces a hierarchical approach to Latent Action Models (LAMs), moving beyond frame-to-frame "primitive" motions to discover temporally extended "skills" from unlabelled video. By integrating a dynamic chunking mechanism, HiLAM segments video trajectories into meaningful sub-tasks, enabling a hierarchical policy that is twice as data-efficient as prior flat-policy baselines on complex, multi-stage robotic tasks.

Positioning: This work addresses a critical bottleneck in robot learning—the scarcity of action-labeled data. It positions itself as a structural upgrade to standard LAMs, shifting the focus from "what is the next move" to "what is the high-level intent."

The "Horizon" Problem in Latent Action Models

Current Latent Action Models (LAMs) function as "action tokenizers" for observation-only data. By looking at two frames, an Inverse Dynamics Model (IDM) can guess the motion ($z_l$) that happened in between. However, like reading a book character-by-character without understanding the words, standard LAMs miss the "skills" (e.g., "opening a drawer" or "picking up a bowl") that span dozens of frames.

Previous attempts to solve this used fixed-length windows (e.g., every 10 frames is a skill). But real-world actions vary: picking up a heavy object takes longer than picking up a light one. HiLAM’s core insight is that skill boundaries should be data-driven, triggered by changes in the underlying dynamics rather than a clock.

Methodology: Hierarchical Logic & Dynamic Chunking

HiLAM operates in two distinct phases: Latent Skill Learning and Hierarchical Policy Learning.

1. The H-Net Architecture

The heartbeat of HiLAM is the H-Net mechanism. It processes a sequence of latent actions and predicts a boundary indicator $b_t$. When the features of consecutive time steps are sufficiently dissimilar, $b_t$ flips to 1, marking the start of a new "chunk" or skill.

2. Training Objectives

To ensure these latent skills are actually representative of physical motion, the model is optimized with a three-part loss:

• ${\mathcal{L}}_{latent}$: A standard next-token prediction loss in the latent space.
• ${\mathcal{L}}_{rec}$: A visual supervision loss where a Forward Dynamics Model (FDM) reconstructs future frames. This tethers the "abstract" skill back to "concrete" visual reality.
• ${\mathcal{L}}_{ratio}$: A regularizer to prevent the model from collapsing (e.g., making every single frame a new skill).

Experimental Performance: Winning at the Long Game

The researchers tested HiLAM on the LIBERO benchmark, focusing on LIBERO-Long, which requires chaining multiple sub-goals.

Data Efficiency

One of the most striking results is HiLAM's performance with limited data. While the SOTA baseline BAKU struggles with only 10% of expert demonstrations (23% success), HiLAM reaches 45%. Essentially, the discovered high-level skills act as a "scaffold," making it much easier for the robot to learn the final task.

Qualitative Boundary Discovery

Does the chunking actually make sense? Visualizations show that HiLAM identifies boundaries that align with human intuition:

• Segment 2: Reaching for the bowl.
• Segment 3: The moment of the "pick" (high dynamics change).
• Segment 4: Moving to the placement area.

Critical Insights & Future Outlook

HiLAM proves that temporal hierarchy is as important as visual representation. By compressing raw sequences into "latent skills," we reduce the planning horizon for the high-level policy, making long-term tasks manageable.

Limitations & Future Work:

• End-to-End Training: Currently, the IDM is pretrained and frozen. Training the whole stack end-to-end could result in "perceptual skills" that are even more robust.
• Language Integration: The authors correctly identify that while motion cues define how a skill is done, language defines what is being requested. Combining HiLAM’s motion-based chunking with linguistic sub-goals is the logical next step for open-ended robotics.

Conclusion: HiLAM is a significant step toward "World Models" that don't just see the world, but understand the hierarchical structure of actions within it.