How does ViSkill reinforce VLM agents with evolving visual-native skills?

Text skills flatten the geometry agents must act on

Skill-augmented agents try to make reinforcement learning more sample-efficient by distilling successful trajectories into reusable strategies and retrieving them for later tasks. The paper argues that most such libraries are text-centric: spatial layouts, relative positions, and action–state correspondences are linearized into language. In a visually grounded environment, that translation can drop the geometric information a strategy depends on, which the authors call a representation gap.

The second gap is about learning. Some recent work does put visual evidence into skill construction or use, but builds and updates those skills separately from policy optimization. The paper's target is therefore not just a better skill format but a coupling: skills should shape the policy, and the policy's own successes should reshape the skills.

A skill card is one image the VLM can read

ViSkill stores each skill as a composite visual skill card. Starting from a successful trajectory, it keeps the original visual observations, adds annotations that clarify spatial and procedural progression, and renders a strategy description distilled by the policy as text inside the same image. The annotated trajectory and the rendered strategy are composed into a single card, which is passed directly to the VLM as visual context.

Each library entry also carries a geometric descriptor, a utility value, and a usage count. The card is the part the model sees; the descriptor and utility are the bookkeeping that decides when the card is worth showing. This separation matters because retrieval in ViSkill is not a language-matching step between the instruction and a written skill description.

Retrieval matches geometry, then weighs past reliability

At the start of an episode, an environment-specific feature extractor maps the initial task configuration to a geometric descriptor capturing relations such as relative positions, distances, and obstacles. For each stored skill, ViSkill computes a score that combines geometric similarity with the skill's historical utility, weighted by a similarity weight of 0.9 in the reported configuration. The top-scoring skill is retrieved only if its score clears a retrieval threshold of 0.8; otherwise the policy proceeds without skill guidance.

The reason to retrieve by geometry rather than by wording is stated directly in the paper: similar instructions can correspond to different spatial layouts that require different strategies. Utility is updated after each episode with an exponential moving average of the binary success indicator, so a skill that repeatedly fails to help loses standing even if its geometry still looks close.

Quality and novelty gates decide what enters the library

Storing every successful trajectory would risk filling the library with suboptimal or redundant skills. After an episode, a successful trajectory is considered for distillation only if it passes two gates. The quality gate requires the trajectory length to be within a task-specific budget, instantiated as the length produced by a task-specific solver. The novelty gate requires that no existing skill already covers the current configuration well, using the best-match score as a proxy for coverage.

For an admitted trajectory, the policy summarizes its own successful interaction into a reusable high-level strategy description, which is rendered with the annotated trajectory into a new card. When the library is full, the skill with the lowest utility–frequency score is evicted. The paper reports that on Sokoban and FrozenLake retrieval becomes nearly ubiquitous as the library grows, while new admissions diminish once common configurations are covered.

Failed trajectories can still earn bounded partial credit

A binary task-success reward gives little signal for a failed trajectory, even when the agent partly followed a useful retrieved skill. ViSkill therefore adds an auxiliary term based on agreement between the executed trajectory and the retrieved reference: it measures how far along the reference trajectory the agent got, relative to the executed and reference lengths, and scales that agreement by a skill-guidance weight.

The term applies only when the episode failed and a skill was retrieved. Because the agreement score lies between 0 and 1 and the guidance weight is set below the terminal success bonus, the auxiliary reward stays smaller than the completion bonus. The authors present this as bounded partial credit that does not distort the global reward ordering. The policy itself is optimized with PPO on the resulting trajectory rewards.

Cold start seeds the library, then the policy takes over

Starting from an empty library means early exploration has no skill guidance, which delays the loop between distillation and policy learning. The optional cold-start mechanism pre-populates the library with seed skills before training: candidate trajectories come from offline task-specific solvers, strategy descriptions come from an external model, and selection uses farthest-point sampling in geometric feature space to maximize initial coverage.

The seeds are only an initialization. Once training begins, all subsequent skills are distilled by the policy from its own successful trajectories. The cold-start library size is set as a fraction of the converged library size from a preliminary run without cold start, and setting that fraction to zero disables external-model initialization entirely. This gives an explicit trade-off between early sample efficiency and independence from external supervision.

0.89 overall success, 0.91 with cold start

The evaluation covers Sokoban, FrozenLake, and PrimitiveSkill, the last spanning five ManiSkill manipulation tasks: Place, Stack, Drawer, Align, and Swap. ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, and the paper reports that this outperforms all evaluated proprietary and open-source baselines while converging faster than standard PPO. These are aggregate figures across the evaluated environments, not a per-task breakdown.

The training setup uses Qwen2.5-VL-3B-Instruct as the policy backbone with PPO, and the same shared optimization and rollout configuration across the three environments, adapting only training horizon, compute, and sequence-length limits. The task-wise PrimitiveSkill curves in the paper show both ViSkill variants reaching full success faster than PPO, with cold start accelerating early learning. The gap between 0.89 and 0.91 is therefore best read as the measured contribution of initialization, not as a change in the skill representation itself.

Where the closed loop stops transferring

Two limits bound how far these results travel. ViSkill is evaluated only on environments supported by the VAGEN framework, and the authors state that whether visual skill cards provide consistent benefits in more complex or open-ended settings remains to be validated. The skill-guided reward also assumes that useful partial progress reaches states represented by the retrieved skill, so it may give limited guidance when valid solutions traverse disjoint state paths.

The practical priority follows from the comparison the paper actually makes. The representation change and the closed loop are what separate ViSkill from text-centric skill libraries, and the cold-start option adds a smaller, separable gain on top. A reader deciding whether the approach fits a new domain should first ask whether that domain's tasks have reusable geometric configurations that a visual card can preserve, and whether successful solutions share state paths with the retrieved references; those two conditions are where the reported mechanism is designed to operate.

What the visual card buys over a written strategy

The paper's central claim is a comparison, not a single score: preserving spatial structure in a form the VLM reads directly, and letting the policy's own successes feed back into that library, is what produces the reported gains. The ablations are described as confirming that visual representations, strategic guidance, and joint optimization each contribute to the positive feedback loop.

That framing sets the decision threshold for readers. If a task's reusable knowledge is mostly verbal procedure, a text skill library may be adequate; if the difficulty lies in relative positions, obstacles, and how actions change the visual state, the card format and geometric retrieval are the parts of ViSkill worth testing first. The 0.89 to 0.91 range applies to the three evaluated environments, and the cold-start fraction is the knob that trades external supervision for faster early learning.

Sources used in this answer