Contact, not pose, is what transfers
The paper's central claim is that contact location and timing describe the task-level physical interaction, while the body surface and joint configuration that produce it depend on the robot. That separation is what allows a human demonstration to be useful to a machine with different limb lengths and proportions. Instead of retargeting whole-body joints, which can move contact points, PanTact measures where and when the human body touches the world and asks the robot to reproduce those contacts with whatever surface fits.
This matters because the alternative pipelines the paper compares against carry only end-effector trajectories or joint poses. Humanoid UMI predicts end-effector actions and tracks only end-effector trajectories; Humanoid UMI++ additionally predicts and continuously tracks knee poses tied to a fixed robot body part. Neither carries an explicit body-contact objective, so arm or leg posture needed for support is left unspecified.
Wearable patches record where and when contact occurs
The data collection interface augments egocentric vision and bimanual UMI grippers with modular contact patches worn on the demonstrator's forearms and knees. Each module is a 44×44 mm unit carrying a single force-sensitive resistor, and a custom 50×50 mm PCB streams up to 16 channels at 1 kHz. Modules snap together magnetically into patches of varying shape and extent, so the same sensing design can cover different demonstrators and body regions.
Contact is binarized against a per-channel adaptive baseline rather than an absolute threshold, because FSR resistance drifts with temperature and mounting pressure. Each patch carries a tracker that maps sensor centers into a world frame, producing an N×4 contact array per timestep: a binary contact state and a 3D position for each sensor. These sequences supervise the policy's contact-goal predictions, and the binary state tells the controller when to track a contact location.
Masking the body closes the viewpoint gap
Whole-body contacts often happen behind the grippers, outside wrist-camera views, so the policy needs an egocentric view with broader scene context. But that view also shows the demonstrator's body, creating an appearance mismatch with the robot, and human-robot height differences shift the viewpoint. PanTact uses a 3D egocentric pointmap and geometrically masks body regions: each gripper frame and contact patch defines a body-facing half-space, and the intersection of those half-spaces approximates the body region, which is masked in the attention layer of the vision encoder.
The policy is a diffusion policy running at 10 Hz that predicts 16-step action chunks. Each step contains a head look-at point, target end-effector poses and gripper widths, and contact goals. A contact goal pairs a binary state with a position; only active goals request that the robot make contact at that point, while inactive goals do not require reproducing the sensor's free-space trajectory. That distinction is what keeps the robot from being over-constrained by body motion that carried no task-relevant contact.
An any-surface objective lets the robot choose its body
The controller converts end-effector trajectories and contact goals into whole-body motion, tracking contact locations only when the predicted state is active. A contact goal is not assigned to a particular robot link. Instead, a smooth proximate-distance objective measures the distance from a commanded contact point to the nearest candidate robot surface, and the controller drives that distance toward zero. Because the goal is not tied to a link, the controller can pick whichever surface suits the robot while maintaining balance.
The paper tests three formulations: upper-body contact-aware inverse kinematics, whole-body IK executed by a dense tracker, and direct sparse contact tracking. In simulation replays of demonstration trajectories, adding contact goals reduced contact surface distance for every controller at little cost to end-effector tracking accuracy. With contact goals enabled, upper-body IK achieved the lowest upper-body end-effector error, while the sparse contact tracker achieved the smallest contact distance and beat the whole-body IK plus dense tracker on both metrics for upper-body evaluation.
Real-robot results across three contact-heavy tasks
All hardware experiments used a 29-DoF Unitree G1 with grippers and cameras mirroring the data collection interface, and every method was trained on the same demonstrations and evaluated from matched test configurations. In grasp-and-embrace, the robot grasps a small object and embraces a bulky one with its arms; success requires moving both simultaneously. PanTact with upper-body IK reached 85% over 20 rollouts and the sparse contact tracker 80%, while Humanoid UMI consistently failed to support the bulky object because its controller tracked only end-effector commands, leaving arm posture unspecified.
In sweep-and-stow, the robot sweeps tall objects into a bin with an arm and then pushes a cabinet closed with a lower-body surface. The end-to-end score is the product of the sweep score and the close-cabinet score, and PanTact with whole-body IK plus dense tracker reached 90.63% over 16 rollouts split across seen and unseen object conditions. Notably, the robot swept with its wrist, forearm, or gripper side and closed the cabinet with either knee or a foot, adapting beyond the demonstrated forearm and left-knee contacts. Humanoid UMI++ often missed objects or knocked them outside the bin and more often approached joint limits, which the authors suggest may stem from human-robot proportion differences when jointly tracking hand and knee poses.
In carry-and-push, the robot carries a box in both arms, walks to a larger floor box, and pushes it more than 20 cm. PanTact with whole-body IK plus dense tracker reached 90% over 10 rollouts, with failures occurring when the robot dropped the carried box, more often for the sparse tracker and Humanoid UMI++. The baseline continuously tracked knee poses and showed substantial knee-pose error, whereas PanTact imposed leg contact objectives only during active contact phases and followed commanded leg contact locations more accurately then.
Where the tradeoff between trackers shows
The controller comparison is not a single ranking. For upper-body tasks, the sparse contact tracker achieved the smallest contact surface distance and also beat whole-body IK plus dense tracker on end-effector error, with reported medians of per-trajectory median errors over 10 grasp-and-embrace trajectories. For whole-body tasks, evaluated over 20 trajectories split equally between sweep-and-stow and carry-and-push, the two trackers traded off: the sparse tracker achieved lower contact distance but higher end-effector error than whole-body IK plus dense tracker.
That split is the practical reading of the paper. If a task depends on precise hand placement, the dense-tracker formulation is the better fit; if the priority is getting body surfaces onto contact targets, the sparse tracker is favored. The paper's own task results follow this pattern, using upper-body IK for grasp-and-embrace and whole-body IK plus dense tracker for the two whole-body tasks.
What the demonstrations and simulator bound
The controller is trained with PPO in 4096 parallel simulated environments using hindsight-labeled rollouts, with domain randomization over masses, friction, and contact-goal perturbations. The paper states this learns whole-body contact realization directly at the cost of simulation training and dependence on the training distribution. The demonstrations themselves are limited in scope: 100 for grasp-and-embrace, 150 for sweep-and-stow, and 110 for carry-and-push, and the hardware evaluation uses 20, 16, and 10 rollouts respectively.
The reported failures also mark the boundary. Occasional grasp-and-embrace failures occur when the gripper misses the small object, and the authors note the grippers continuing to reach forward until approaching arm joint limits, potentially reflecting accumulated tracking error that takes observations outside the policy's training distribution. Occasional partial cabinet closures occur when the controller prioritizes balance over reaching the contact target. These are contact-realization and distribution limits, not evidence that the approach scales to arbitrary tasks or embodiments.
Why active contact goals are the priority
The most decision-relevant distinction in this paper is between active and inactive contact goals. Only active goals request contact at a predicted position; inactive goals do not require reproducing the demonstrator's free-space body motion. That is why the robot can close a cabinet with a knee or a foot rather than the demonstrated left knee, and why it can sweep with a wrist or gripper side rather than the demonstrated forearm. The controller is free to choose a body configuration that satisfies the task-critical contacts while staying balanced.
For readers comparing whole-body manipulation approaches, the paper's priority is therefore not maximum pose fidelity but contact fidelity under robot-specific constraints. The evidence supporting that priority is the controller comparison, where adding contact goals reduced contact surface distance for every controller at little cost to end-effector accuracy, and the task results, where pose-only baselines failed specifically on the contacts they left unspecified: arm support in grasp-and-embrace, arm regularization in sweep-and-stow, and sustained arm support during locomotion in carry-and-push.
Sources used in this answer
PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface
PanTact learns whole-body humanoid manipulation from robot-free human demonstrations by transferring contact goals rather than body poses, reaching 85% on grasp-and-embrace, 90.63% end-to-end on sweep-and-stow, and 90% on carry-and-push on a 29-DoF Unitree G1 without additional body contact sensors.