[CVPR 2025] ViterbiPlanNet: Efficiency Reigns Supreme via Differentiable Procedural Knowledge

TL;DR

ViterbiPlanNet challenges the trend of using massive LLMs and Diffusion models for video procedural planning. By introducing a Differentiable Viterbi Layer (DVL), the authors bake structural constraints directly into the training process. The result? A model with only 5M parameters that beats 1B+ parameter heavyweights in predicting complex action sequences from visual start and goal states.

The "Memorization" Trap in Procedural Planning

Procedural planning — the ability to look at a "before" and "after" image and infer the steps in between — is a hallmark of intelligent agents. Until now, the state-of-the-art relied on implicit learning: feeding Transformers or Diffusion models thousands of sequences and hoping they "memorize" that you must "add turkey" after "placing bread."

This is fundamentally flawed for two reasons:

1. Data Inefficiency: It takes massive amounts of data to learn basic physics/logic.
2. Structural Fragility: Models often "hallucinate" impossible transitions because they lack an explicit understanding of the task's graph.

Methodology: Making Viterbi Differentiable

The core Innovation of ViterbiPlanNet is the Differentiable Viterbi Layer (DVL). While classical Viterbi is a "hard" algorithm (using max and argmax), the authors apply smooth relaxations.

How it Works:

1. PKG Extraction: A Procedural Knowledge Graph (PKG) is built from training data co-occurrences.
2. Emission Learning: Instead of predicting a sequence, the neural network only predicts the "compatibility" (emissions) of an action with a visual state.
3. End-to-End Decoding: The DVL takes these emissions and the PKG to "solve" the most likely path. Because the layer is differentiable, the error from the final plan flows all the way back to the visual encoder.

This setup ensures the model never has to learn how to plan — it only has to learn what actions look like, while the DVL handles the logic.

Experiments: David vs. Goliath

The results are a masterclass in parameter efficiency. ViterbiPlanNet was tested against foundations like Gemini 2.5 Pro and MTID (1B parameters).

Key Breakthroughs:

• Success Rate (SR): Consistently higher than all re-benchmarked methods.
• Sample Efficiency: As shown in the ablation studies, ViterbiPlanNet reaches peak performance with significantly less training data because it doesn't waste capacity "re-learning" the graph structure.
• Cross-Horizon Robustness: Unlike previous models that fail when asked to plan for a shorter length than they were trained on (e.g., training on T=6, testing on T=3), ViterbiPlanNet remains robust because its logic is grounded in the PKG, not fixed-length sequence memorization.

Deep Insight: Decoupling Emissions from Transitions

A fascinating qualitative finding is that ViterbiPlanNet learns "cleaner" emissions. In standard models, the network tries to bake the probability of the next action into the current prediction. ViterbiPlanNet decouples this: the neural network focuses purely on visual evidence, while the DVL handles the "legal moves." This results in smoother, more interpretable latent spaces.

Conclusion

ViterbiPlanNet proves that inductive biases matter. In an era of "just add more layers," this paper serves as a reminder that a well-placed mathematical prior (like a differentiable Viterbi algorithm) can outperform models 100x its size. This is particularly vital for the future of wearable AI and robotics, where on-device, efficient reasoning is the gateway to real-world utility.