DexForge adapts human contacts instead of copying them
The paper's central claim is that a human demonstration's contacts should not be transferred directly to a robot with a different hand shape. Instead, DexForge first adapts those contacts to the target robot, then uses the adapted contacts to guide both kinematic retargeting and dynamic control optimization. The stated motivation is that contacts from a human demonstration may be unreachable or unstable for a robot hand, and that a penetration-free robot trajectory with geometrically appropriate contacts may still fail under contact dynamics and actuation constraints.
This reframes the retargeting problem. Rather than treating the video as a pose target to be matched, DexForge treats it as a source of interaction structure that must be re-established on the robot before any dynamics optimization begins. The paper describes the pipeline as progressively establishing the physical structure required for robot execution: reconstruct geometry and motion, adapt contacts and build a collision-constrained kinematic reference, then realize that reference by optimizing controls through a differentiable simulator.
Spherical Gaussians carry both appearance and collision geometry
DexForge reconstructs object assets from multi-view RGB-D video as spherical Gaussian models, so a single representation supplies visual appearance for pose estimation and geometry for collision checking. The paper contrasts this with ContactGaussian-WM, which also unifies appearance and collision geometry with spherical Gaussians but optimizes geometry and appearance separately. DexForge's stated concern is that image-based initialization does not directly constrain interior distances, potentially yielding unreliable penetration-depth estimates, and that a later color-fitting stage holding biased geometry fixed can limit appearance accuracy.
Its fix is depth-derived distance supervision. The Gaussian collision distance field at a query point is defined as the minimum over primitives of the point-to-center distance minus the primitive's collision radius, with the radius set to three times the Gaussian scale. A reference signed distance field built from multi-view depth maps supervises this field, and the distance-matching loss is combined with the standard 3DGS objective so geometry and color are optimized jointly. The reconstructed model is then used with GS-Pose to recover the object pose trajectory, while MANO hand motion is recovered via HaMeR predictions and camera calibration, placing both trajectories in a common coordinate frame.
Kinematic and dynamics retargeting solve different problems in sequence
The paper formulates retargeting as two sequential constrained optimization problems with complementary roles. Kinematic retargeting seeks a robot reference trajectory that matches the demonstrated motion through keypoint fitting, using a correspondence set to fit forward-kinematics robot keypoint positions to their MANO counterparts, subject to configuration and collision constraints. Dynamics retargeting then optimizes robot controls to track the reference state trajectory, starting from the reference initial state, subject to the simulated dynamics and admissible state and control constraints.
The distinction matters because the two stages answer different questions. Kinematic retargeting asks whether a geometrically valid, penetration-free robot pose sequence exists that preserves the demonstrated contact structure. Dynamics retargeting asks whether that sequence can actually be executed under contact dynamics and actuation limits. The paper notes a trade-off in physics-based optimization: sampling-based methods can handle complex contact changes but may need many samples for very accurate trajectories, while gradient-based optimization refines trajectories more directly but is sensitive to initialization and local minima. DexForge's response is to use robot-adapted contacts as a good kinematic reference and as continued guidance during gradient-based dynamics optimization.
Contact-aware retargeting removes penetration and tracks refined anchors
In kinematic retargeting, the paper compares against SPIDER's IK-based stage on all selected DexYCB and HOT3D sequences across seven hands. The baseline fits MANO-derived fingertip positions and wrist orientation without considering hand-object interactions or stable contacts. Both methods are evaluated with dataset-provided ground-truth trajectories, and DexForge is additionally evaluated on its own reconstructed trajectories.
The reported kinematic errors are in millimeters. On DexYCB with ground-truth inputs, the IK baseline shows self-collision error of 4.73 and hand-object penetration of 6.37, while contact-aware retargeting reports 0.00 for both. Raw anchor tracking error drops from 23.30 to 9.20, and refined anchor tracking error from 24.93 to 3.92. On HOT3D with ground-truth inputs, IK shows 5.37 self-collision and 8.80 hand-object penetration, versus 0.00 and 0.01 for contact-aware retargeting; raw anchor error falls from 23.72 to 12.87 and refined anchor error from 24.01 to 5.59. The paper states that comparable results with its reconstructions suggest limited impact on kinematic retargeting performance, and that the method reduces penetration while adapting contacts to the robot and preserving the demonstrated contact structure.
Gradient-based force-aware refinement drives the largest gains
For dynamics retargeting, the paper compares DexForge with sampling-based SPIDER across seven hands in ComFree-GS. To isolate the contribution of the dynamics stage, it adds a controlled comparison called Contact-aware+SPIDER, which replaces SPIDER's IK stage with DexForge's contact-aware kinematic stage and supplies DexForge's contact guidance to SPIDER's optimizer. All methods use dataset-provided ground-truth trajectories, and DexForge is additionally evaluated on its reconstructions. Position and rotation errors are computed over successful trajectories only, averaged over five seeds.
On DexYCB, IK+SPIDER reports 8.21 mm position error, 4.22 degrees rotation error, and 24.9% success; Contact-aware+SPIDER reports 7.57 mm, 3.37 degrees, and 50.6%; DexForge reports 2.73 mm, 0.93 degrees, and 77.4%. On HOT3D, IK+SPIDER reports 7.55 mm, 4.70 degrees, and 0.4% success; Contact-aware+SPIDER reports 7.07 mm, 3.29 degrees, and 0.6%; DexForge reports 5.01 mm, 1.34 degrees, and 34.9%. The paper attributes the remaining gap to gradient-based force-aware refinement, since the kinematic stage and contact guidance are held constant in the Contact-aware+SPIDER comparison. It notes the gap is largest on HOT3D, where objects are typically in hand from the first frame and SPIDER mostly fails through object drops, suggesting simulation gradients help sustain precise in-hand contact. DexForge with reconstructed inputs reports 74.1% success on DexYCB and 31.2% on HOT3D.
System identification decides whether trajectories survive transfer
The paper evaluates open-loop transfer from ComFree-GS to MuJoCo-CPU on the same subsets as the ablation study, with all methods using reconstructed trajectories. The target uses mesh-based collision detection and object-specific physical parameters, while ComFree-GS uses Gaussian collisions and shared initial parameters. Per-sequence system identification calibrates physical parameters and a collision-distance offset to reduce dynamics mismatch and compensate for residual geometric error; controls are then reoptimized in the calibrated simulator and replayed open-loop. Transfer counts as successful when mean tracking errors satisfy a position threshold of 0.02 m and a rotation threshold of 0.2 rad.
The calibrated parameters are object mass, inertia tensor, contact friction coefficient, and collision-distance offset. Because parameter coupling may produce a multimodal identification landscape, the paper uses TSMC to combine particle-based exploration with gradient-informed refinement, and compares it against gradient-only and sampling-only identification. On DexYCB, DexForge transfer success rises from 25.7% before identification to 36.6% with gradient-only, 53.1% with sampling-only, and 58.9% with TSMC. On HOT3D, DexForge rises from 9.1% to 13.1%, 26.9%, and 33.1% respectively. The same ordering holds for the two baselines, so the choice of identification method changes transfer outcomes by tens of percentage points.
Real-robot replay measures executability, not tracking fidelity
For sim-to-real open-loop transfer, the paper lacks accurate object-pose measurements and therefore evaluates transfer without system identification. For five YCB objects, it runs the full retargeting pipeline on its reconstructions with known physical parameters, then replays the resulting controls open-loop on a Unitree G1 equipped with a LEAP Hand v1 from simulation-matched initial positions. Over 10 trials per object, executions achieve an 80% success rate in stable grasp-and-hold.
The paper explicitly states that this measures executability rather than tracking fidelity. That distinction is the key boundary on the real-robot result: an 80% stable grasp-and-hold rate does not establish that the demonstrated object motion and orientation were reproduced on hardware, and it does not carry the position and rotation error figures reported for simulation. The real-robot evaluation also omits system identification, which the sim-to-sim results show materially affects transfer success.
The strongest evidence is the controlled dynamics comparison
The most decision-relevant result is the controlled comparison in which the kinematic stage and contact guidance are held constant: Contact-aware+SPIDER versus DexForge. That comparison isolates the gradient-based force-aware dynamics refinement, and it is where the largest success-rate and tracking-error differences appear, especially on HOT3D where the sampling baseline nearly fails. The kinematic results, by contrast, show that contact-aware retargeting already removes penetration and improves anchor tracking, but they do not by themselves establish executability.
The paper's own framing of the trade-off is the practical threshold to carry forward: sampling-based optimization handles complex contact changes but may need many samples for accuracy, while gradient-based optimization refines directly but is sensitive to initialization and local minima. DexForge's answer is to use robot-adapted contacts as initialization and continued guidance. The transfer results add a second threshold: whether a trajectory survives a change of simulator depends heavily on how physical parameters are identified, with TSMC outperforming both gradient-only and sampling-only identification on both datasets. Readers weighing this work should therefore separate three claims that the paper keeps distinct: penetration-free kinematic references, high success and low tracking error in the differentiable simulator, and open-loop executability on hardware measured only as stable grasp-and-hold.
Sources used in this answer
DexForge: High-Fidelity Physics-Informed Dexterous Retargeting
DexForge converts human video demonstrations into robot hand trajectories by adapting human contacts to the target robot and guiding both kinematic and gradient-based dynamics retargeting, reporting success-rate gains of roughly 35-53 percentage points over the baseline across 130 DexYCB and HOT3D demonstrations and seven dexterous hands.