CoInteract: Solving the Physical Consistency Gap in Human-Object Interaction Video Synthesis

Executive Summary

TL;DR: CoInteract is a novel framework designed to generate realistic videos of humans interacting with objects (e.g., product demonstrations). By integrating a Human-Aware MoE and a Spatially-Structured Co-Generation strategy into a Diffusion Transformer (DiT), it eliminates common artifacts like "melted fingers" and hands passing through solid objects.

Background: Within the landscape of generative AI, we have moved from talking heads to full-body motion. However, Human-Object Interaction (HOI) represents the next frontier. CoInteract shifts the paradigm from "appearance-only" generation to a "structure-aware" engine, establishing a new SOTA for physically plausible video synthesis.

Problem & Motivation: The RGB-Centric Limitation

Existing video diffusion models treat every pixel equally. While this works for landscapes, it fails for HOI because:

• Physical Blindness: Without an explicit notion of 3D boundaries, models often allow hands to penetrate objects (interpenetration).
• Structural Fragility: Anatomical details in hands and faces require higher-order precision that global attention mechanisms often smooth over, leading to blurred or anatomically impossible features.

The authors argue that the model must not only "see" pixels but also "understand" the underlying interaction geometry.

Methodology: The Dual-Stream "Scaffold"

To bridge the gap between appearance and physics, CoInteract introduces two core innovations:

1. Spatially-Structured Co-Generation

The model follows a dual-stream training paradigm. While one stream focuses on RGB Appearance, an auxiliary HOI Structure Stream focuses on the "scaffold"—a 3D-projected human mesh fused with object masks.

• Cross-Stream Learning: Using a shared DiT backbone, the RGB tokens attend to the HOI structural tokens.
• Asymmetric Co-Attention: This is the "secret sauce." During training, the HOI stream regularizes the shared weights. At inference, the HOI branch is discarded, allowing the RGB stream to benefit from the learned physics with zero performance penalty.

2. Human-Aware Mixture-of-Experts (MoE)

Generic Feed-Forward Networks (FFNs) in Transformers often struggle to balance global background details with local hand/face intricacies. CoInteract uses a Spatially Supervised Router to send specific tokens to:

• Head Experts: For identity preservation.
• Hand Experts: For precise finger articulation.
• Base Experts: For everything else.

Experiments & Results: Superior Realism

CoInteract was tested against 6 major baselines (including SkyReels-V3 and InteractAvatar).

Key Findings:

• HOI Plausibility: In VLM-QA (evaluated by Gemini-1.5-Pro), CoInteract scored 0.72, far exceeding the 0.44-0.62 range of competitors.
• Fine-Grained Quality: Hand Quality (HQ) improved to 0.724, indicating much more stable skeletal structures.
• Inference Efficiency: Despite the complex training, the inference cost is only 1.04x compared to a baseline model, whereas keeping the structure stream active would have cost 4.13x.

Visual evidence (Fig. 7) shows that without Co-Generation, hands clip through objects; without MoE, hands become blurry and lose their shape.

Critical Analysis & Conclusion

Takeaway: CoInteract proves that specialized "expert" capacity and structural auxiliary supervision are the keys to professional-grade video generation. By separating the concept of physical structure from the rendering of RGB pixels during training, we can achieve high-fidelity output without the computational drag of 3D simulation at runtime.

Limitations: The model relies on the quality of the initial 3D mesh recovery and object masks. If the preprocessing tools (like SAM3) fail on highly occluded objects, the "scaffold" might be inaccurate.

Future Work: This "Co-Generation" strategy could likely be extended to other auxiliary modalities like optical flow or depth, further grounding AI video generation in real-world physics.