A benchmark that splits understanding from responding
PsycheMME is a video question-answering benchmark built around a specific question: can multimodal foundation models infer human states and needs from observable cues, and then make interaction judgments suited to the context? The paper argues that recognizing actions is not enough, because a model must also interpret social cues and decide what to do next.
Its organizing idea comes from Social Information Processing and the Perception-Action Cycle, which the authors use to separate cue encoding and interpretation from goal clarification and response selection. That separation is the benchmark's main design choice: instead of one accuracy number, it reports where a model fails between observing a social situation and choosing a response.
The benchmark contains 1,000 questions drawn from 630 video episodes, spanning eight capabilities. Because capability counts are uneven, the paper uses macro averaging rather than letting the largest category dominate the reported score.
Eight capabilities across three diagnostic domains
The eight capabilities fall into three groups. Human understanding covers Social Grounding (G), which links speech, gaze, gestures, and actions to the right participants; Psychological Understanding (W), which infers states and their possible causes; and Need Inference (N), which identifies desired support and distinguishes explicit requests from inferred needs and willingness to accept help.
Interaction decisions cover Timing (T), which judges whether to respond now, keep listening, wait, or refrain; Empathic Verbal Responding (S), which selects response content, strategy, and tone; and Embodied Social Action (A), which selects physical or nonverbal behavior such as gaze, distance, assistance, or retreat.
Context and constraints cover Interaction and Profile Memory (M), which applies supplied history and individual preferences, and Risk, Boundary, and Autonomy Regulation (R), which checks a proposed intervention against harm, consent, boundaries, and autonomy. The distinction matters because N asks what support is needed, while R asks whether the proposed response is permissible.
Why a correct answer can still earn partial credit
The paper's central measurement is the Reasoning-Verified protocol. Instead of scoring only the final answer, it weights answer credit by how well an explicit rationale is supported by annotated cues and by whether that rationale is logically consistent. A model that selects the right option for the wrong reason is therefore not credited the same as one that selects it with supported reasoning.
This design responds to an observation the authors report: a correct response strategy can be justified with cues from the wrong time window. Aggregate accuracy would hide that failure, while scoring answers alongside rationales locates it.
The benchmark also mixes formats: 890 choice questions, including single-select and multi-select, and 110 open-ended questions. Open questions matter for a separate reason, because they expose differences in answer completeness that a choice format cannot show.
Independent answering keeps proposed answers from leaking
Questions were generated through a multi-agent pipeline with deliberate information isolation. A Question Agent proposes a question, candidate answer, options, and supporting cues. An Answer Agent then re-examines the video using only the question, options, and public context, without access to the proposed answer, cues, or private rationale, and can flag ambiguity or insufficient support. A Critic Agent checks both paths for answerability, cue support, and logical coherence.
Separating proposal from independent answering reduces the risk of accepting a proposal because its own justification sounds plausible. Items that still fail after two refinement rounds are discarded, and a shared state with role-based access control enforces the information isolation.
Five researchers with psychology backgrounds then independently reviewed question validity, answer correctness, and psychological consistency. Only unanimously approved items entered the frozen benchmark, and fine-tuning data were kept disjoint from benchmark items.
Timing is where models lose the most credit
Across the 17 evaluated models, Timing had the lowest Verified scores and the lowest credit retention. Credit retention refers to how much of the raw answer credit survives verification of the rationale, so a low value means correct Timing answers were often weakly supported by the cues the model cited.
The paper reports that models made weaker support judgments specifically for correct Timing answers. This is a narrower claim than saying models cannot tell time: the failure appears in the justification attached to an answer, not only in whether the answer was right.
The adaptation experiments point in the same direction. On the same 1,000 questions, Qwen3.5-4B Base scored 47.68 raw and 37.71 verified; supervised fine-tuning reached 72.58 raw and 52.13 verified; GRPO starting from that checkpoint reached 73.19 raw and 52.62 verified. SFT improved several capabilities, but Timing remained below the Base score after both adaptation stages. The authors note this comparison does not isolate temporal sampling effects from the training objectives.
Similar Need scores can hide different answer completeness
A second diagnostic result concerns open questions. Models with similar Need scores still differed in answer completeness on open-ended items, meaning a shared score on need inference did not imply that models produced equally complete answers.
This is why the benchmark reports answer completeness separately rather than folding it into a single capability score. A model can identify the relevant need and still give a partial answer, and the choice format would not reveal that gap.
The paper also reports that voting can override correct minority answers, which matters for anyone aggregating multiple samples from a model. More samples do not automatically produce a better answer when the majority is wrong.
Context controls and adaptation change the picture
Input controls and history controls test how much performance depends on the supplied context. The paper reports that these controls reveal contextual dependence, and it examines modality and single-image variants alongside history interventions.
The adaptation comparison is separate from the 17-model foundation evaluation. Both SFT and GRPO update LoRA parameters in the language backbone of Qwen3.5-4B while freezing the visual encoder and multimodal aligner. GRPO's binary reward requires both exact answer correctness and compliance with response requirements, while Evidence Support and Logical Integrity scores are used only for evaluation.
Because the adaptation comparison uses its own checkpoints and a screening set distinct from the final evaluation set, its numbers should be read as a within-model adaptation result rather than as a ranking against the 17 foundation models.
Recorded interaction is not live interaction
The benchmark evaluates human-centered reasoning in recorded social interactions. Real-time interactive settings remain unevaluated, and the authors state that evaluation remains bounded by the represented social and cultural contexts.
That boundary matters most for Timing. Judging when to speak in a recorded clip is not the same as acting on a live stream, so the low Timing scores should be read as evidence about offline judgment rather than a measurement of conversational behavior in deployment.
The authors propose expanding contextual and cultural diversity, exploring streaming and interactive evaluation, and adding finer-grained human assessments of explanation quality. Until then, the benchmark's value lies in its capability-level diagnosis, not in a single leaderboard position.
Read the verified profile before the raw score
For anyone comparing models on this benchmark, the paper's own priority is clear: the gap between raw and verified scores, and the capability-level profile behind it, carries more diagnostic information than the aggregate. A model with a strong raw score but weak credit retention is answering correctly without adequately supporting its answers.
The most decision-relevant split is between understanding and response selection. A model can ground participants and infer states while still failing on Timing, and it can match another model on Need while producing less complete open answers. Those are different failures with different remedies.
The paper's specific warning is that Timing is the capability where correct answers are least well justified, and where adaptation did not close the gap. That is the first place to look when a model appears socially competent on aggregate accuracy.
Sources used in this answer
PsycheMME: A Psychology-Informed Benchmark for Social Understanding and Interaction in Videos
Across 17 models, Timing had the lowest Verified scores and credit retention, and models with similar Need scores still differed in answer completeness on open questions.