How can multimodal large language models enable training-free HOI detection?

Training-free HOI detection still needs grounded participants

Human-object interaction detection localizes human-object pairs and recognizes the interaction between them, usually written as a human-action-object triplet. Supervised detectors learn this from HOI annotations, which the paper argues limits scalability to the compositional and long-tailed interaction space of real scenes. Training-free paradigms instead exploit frozen pretrained models directly, using their visual-semantic knowledge without task-specific optimization.

The paper's diagnosis is not that frozen MLLMs lack knowledge, but that existing training-free pipelines invoke it through loosely coordinated inference stages. Because interaction hypotheses do not steer visual exploration, key participants are left overlooked and local ambiguities unresolved. The authors add a second failure mode: propagating early semantic assumptions through later visual grounding and relation prediction induces self-reinforcing semantic circularity. HarnessHOI is proposed as a fix for both, not as a new backbone.

Interaction hypotheses steer where the model looks next

The interaction-guided perception mechanism projects emerging interaction hypotheses back into visual space. Instead of a single global pass, it uses action and role hypotheses to discover relevant participants and to select regions for closer observation, so that missing participants can be found and ambiguous evidence refined through targeted observation.

The paper separates two contributions inside this mechanism. Participant discovery adds identities that global localization missed; targeted observation adds local crops chosen according to the interaction hypothesis. Both are tested against controls that keep the same image, initial participants, generation budget, and downstream observation, geometric adjudication, and relation verification steps. The repeated-localization control instead asks directly for additional visible participants associated with the designated humans, and the non-directed control samples crop centers uniformly from valid image positions with a fixed seed. The reported gains are attributed to interaction semantics guiding search and region selection rather than to extra computation alone.

Geometric adjudication builds one grounding shared across roles

The relation-agnostic geometric adjudication module reconciles multi-source evidence to establish a unified spatial basis for grounded interaction reasoning across multiple actions and semantic roles. In practice this means the same adjudicated instance groundings are reused when verifying different actions and roles, rather than re-deriving a separate spatial basis per relation.

This design choice is what the paper contrasts with relation-conditioned adjudication and proposal-conditioned verification, which score candidates under a specific relation or proposal. The paper reports that the complete harness outperforms both controlled alternatives on HICO-DET Full and on V-COCO S1 and S2, with the largest single-stage gains coming from participant discovery and relation verification. The interpretation offered is that a shared spatial basis avoids letting an early relation guess dictate which evidence is admissible later.

HICO-DET and V-COCO results under one frozen backbone

On HICO-DET Default evaluation, HarnessHOI reports 33.20 mAP Full, 41.63 Rare, and 30.68 Non-Rare; under Known Object evaluation it reports 36.85 Full, 44.20 Rare, and 34.65 Non-Rare. On V-COCO it reports 44.65 role AP under S1 and 49.70 under S2, averaged over 25 roles. All variants in these comparisons use the same frozen backbone and evaluator.

The paper also reports a common-evaluator comparison against released AgentHOI predictions, which use GPT-4o and GroundingDINO while HarnessHOI uses Qwen3.5-9B. Under identical matching and AP computation, HarnessHOI leads by 3.52 points on Default Full, 1.14 on Default Rare, and 4.23 on Default Non-Rare, and by 3.14, 0.81, and 3.83 points respectively under Known Object evaluation. The release covers 9,546 test images, including 23 without boxes, and the remaining 112 official test images receive empty predictions; both methods are scored over the complete test set with identical inclusive-IoU, matching, and sorting conventions.

Unseen-class gains hold across four open-vocabulary splits

The class-partition analysis evaluates the same HarnessHOI predictions under four standard partitions. On Unseen classes it reports 40.15 under RF-UC, 33.64 under NF-UC, 35.45 under UO, and 33.87 under UV, against AgentHOI common-evaluator Unseen values of 39.11, 28.56, 33.97, and 30.13. The paper states that gains extend to Unseen classes in all four open-vocabulary settings.

The same table shows HarnessHOI's Seen values of 31.46, 33.09, 32.75, and 33.09, and Full values of 33.20 in every partition. That pattern matters for reading the headline number: the Full figure is a single aggregate, while the Unseen and Seen columns describe different class subsets, so the Unseen advantage should not be read as a uniform improvement across all classes. The paper also reports that positive-interaction and no-interaction classes are assessed through separate visual questions, with Default mAP of 36.84% over the 520 positive-interaction classes and 9.55% over the 80 no-interaction classes, giving the class-weighted Full mAP of 33.20%.

Ablations show discovery and verification carry most of the gain

The harness ablations isolate each stage. Removing discovered participants drops HICO-DET Default Full to 32.55 and V-COCO to 44.02 S1 / 48.93 S2; removing targeted observation gives 32.80 and 44.33 / 49.21; replacing geometric adjudication with first-candidate geometry gives 32.95 and 44.38 / 49.48. Direct HOI generation falls to 15.62 on HICO-DET and 16.41 / 21.20 on V-COCO, and an initial-instance baseline without the harness reaches 31.45 and 42.90 / 47.20.

The paper's stage-wise gain figure reports the complete harness over each controlled alternative: +0.75 for participant discovery, +0.50 for targeted observation, +0.41 for geometric adjudication, and +0.70 for relation verification on HICO-DET Full, with comparable or larger values on V-COCO S1 and S2. Coverage accounting shows discovery adds matching groundings for 571 HICO-DET interactions and 217 V-COCO roles, local observation extends coverage by a further 12 and 23 annotations, and the final selected groundings cover 22,895 HICO-DET interactions and 14,473 V-COCO roles. These are coverage counts of annotations, not mAP, and should not be substituted for the detection metrics.

Inference cost and the boundary of the training-free claim

The harness is not free at inference time. Per image on HICO-DET it averages 2.51 generative responses and 563.29 output tokens, with 119.81 scoring requests on average and 470 at the 95th percentile, costing 13.26 GPU-worker seconds per image; V-COCO averages 2.78 responses, 615.26 output tokens, 311.27 scoring requests with 1,011 at P95, and 22.07 GPU-worker seconds per image. Scoring input cache hits are 88.43% and 89.93% respectively. The paper measures this with one model replica per GPU, four active images per replica, up to 32 concurrent requests per GPU, and four GPUs.

The training-free claim is about optimization, not about resources: HarnessHOI uses one frozen Qwen3.5-9B for participant acquisition, geometric adjudication, and relation verification, with no auxiliary models or retrieved visual support sets, whereas the compared AgentHOI by Lei et al. uses GPT-4o or Qwen2.5-VL-72B with GroundingDINO and DA-HOI (TF) uses R50 DETR proposals. The evidence is limited to HICO-DET and V-COCO, both standard public benchmarks, and the paper is under review, so the reported state-of-the-art ranking among training-free methods reflects the compared set and the authors' evaluation protocol rather than an exhaustive survey.

Compare training-free methods on the same evaluator and split

The most decision-relevant lesson from this paper is procedural. HarnessHOI's advantage over AgentHOI is reported both as source-reported numbers and as a common-evaluator comparison over the complete test set with identical inclusive-IoU, matching, and sorting conventions, and the two do not coincide: AgentHOI's reported Unseen RF-UC value is 38.64 while its common-evaluator value is 39.11. Any comparison of training-free HOI methods should therefore fix the evaluator, the class partition, and the treatment of images without boxes before reading the gap.

Within that controlled setting, the paper's own priority ordering is clear: participant discovery and relation verification contribute the largest single-stage gains, geometric adjudication contributes the smallest, and the shared-grounding design is what lets one frozen model serve exploration and verification without auxiliary detectors. Readers weighing this approach should check the Unseen-versus-Seen split and the per-image GPU-worker cost alongside the headline mAP, because the Full aggregate of 33.20 hides both the open-vocabulary advantage and the inference budget that produces it.

Sources used in this answer