AnyLift: Scaling 3D Motion and HOI Reconstruction via 2D Diffusion

TL;DR

AnyLift is a breakthrough framework that allows researchers to "lift" 3D human motion and human-object interactions (HOI) directly from unconstrained monocular Internet videos. By utilizing a two-stage diffusion process and a camera-conditioned architecture, it overcomes the diversity limitations of traditional MoCap datasets and the geometric constraints of static-camera models.

Background & Positioning

Reliable 3D human behavior data is the "fuel" for character animation and humanoid robotics. However, high-quality MoCap is expensive and limited to lab settings. While moving to "in-the-wild" video is the obvious solution, the challenges are immense: dynamic cameras, self-occlusion, and the absence of 3D ground truth. AnyLift positions itself as a scalable bridge, using the generative power of 2D diffusion to learn 3D priors from the vast diversity of YouTube.

The Problem: The "Lab-to-Wild" Gap

Most SOTA models are trained on datasets like AMASS. When they encounter a gymnast or a martial artist on the web, they fail because:

1. Distribution Shift: MoCap doesn't cover extreme athletic poses.
2. Global Trajectory: Static-camera models can't distinguish between human movement and camera movement.
3. Viewpoint Sparsity: Internet videos are usually filmed from a few "sweet-spot" angles, making it hard for models to learn 3D structure from all sides.

Methodology: The AnyLift Framework

AnyLift introduces a two-stage pipeline that emphasizes Multi-View Consistency and Camera Awareness.

1. Multi-View Synthetic Data Generation

Instead of needing 3D labels, AnyLift uses a Single-View 2D Motion Diffusion Model. It is unique because it is conditioned on:

• Camera Trajectories: To decouple root movement from camera panning.
• Epipolar Lines: To enforce geometric constraints between synthesized views.

2. Hybrid Data-Source Training

To solve the "Limited Viewpoint" problem, the authors mix two data streams:

• Real 2D Keypoints: High diversity but limited angles.
• Reprojected 3D Poses: Lower diversity but allows for arbitrary synthetic viewpoints.

Figure 1: The AnyLift Pipeline. Stage 1 generates the consistency needed for Stage 2 to learn a robust 2D-to-3D lifting prior.

Experiments: Dominating the Wild

AnyLift was tested on "at-scale" tasks involving gymnastics and martial arts, where traditional methods like WHAM or SMPLify often result in "foot sliding" or ground penetration.

Performance Comparison

In terms of Root Translation (Troot)—the metric most sensitive to camera movement—AnyLift showed a massive improvement, achieving errors of 64.2mm compared to 164.3mm for MoCap-bound methods like WHAM.

Figure 2: Qualitative results on the BEHAVE dataset. Note how AnyLift maintains contact between the human and objects (chairs/tables) without the "floating" or "penetration" artifacts seen in VisTracker.

Critical Insight: Why does it work?

The secret sauce is the Score Distillation Sampling (SDS) within a camera-aware manifold. By phrasing 3D reconstruction as a "multi-view 2D generation" problem, the model uses the diffusion prior to "guess" missing information (depth, occlusion) while remaining anchored by the physical constraints of epipolar geometry.

Summary & Future Outlook

Takeaway: AnyLift proves that we don't necessarily need more 3D data; we need better ways to exploit the 2D data we already have.

Limitations: The model still requires a category-specific "object mesh" for HOI tasks and assumes centered human tracking. Future work could integrate "Open-Vocabulary" object detection to enable a truly universal lifting engine for any object encountered in the wild.

Conclusion: This work paves the way for building "Behavioral Foundations Models" that can learn the full spectrum of human interaction directly from the digital footprint of human culture: the Internet.