[ICASSP] MEMI: Breaking the One-to-One Constraint in Multimodal Speaker Personality Recognition

TL;DR

Understanding someone’s personality from a video is a complex task where timing is everything—but not always in a synchronized way. This paper introduces MEMI (Multimodal Explicit Many-to-Many Interactions), a model that allows text, audio, and video features to interact across any time step. By moving away from rigid "one-to-one" time-step alignment, the researchers achieved significant performance gains across 12 distinct personality traits.

Problem & Motivation: The "Joke" Dilemma

In human communication, a serious face might precede a burst of laughter, or a gentle tone might contrast with a harsh word five seconds later. Traditional models typically use implicit one-to-one interactions, where features from different modalities are fused only at the exact same timestamp ($t$).

This approach has two fatal flaws:

1. Implicit Propagation: Information is buried in the hidden states of uni-directional LSTMs, making it hard to maintain long-range dependencies in videos that can exceed 600 words.
2. Temporal Rigidity: It ignores interactions between a gesture at time $t_1$ and a word at $t_2$, which are often crucial for identifying traits like "Humorous" or "Nervous."

Fig 1. Traditional implicit one-to-one vs. the proposed explicit many-to-many architecture.

Methodology: The Core Architecture

The proposed MEMI framework follows a structured pipeline of extraction, encoding, and explicit interaction:

1. Feature Extraction & Encoding

The model extracts manual transcripts (GloVe), visual cues (Action Units, Gaze), and acoustic features (COVAREP). Unlike previous works, it uses a Bi-directional LSTM (Bi-LSTM) to encode these sequences, ensuring that the model "sees" the entire context of the monologue from both directions, preventing information decay.

2. Explicit Many-to-Many Attention

The engine of MEMIs is its attention mechanism. Instead of simple concatenation, the model calculates:

• Intra-modal interactions: How the beginning of a sentence relates to its end.
• Bi-modal interactions: How text relates to visual sequences across all time steps ($ToV$).
• Tri-modal interactions: Complex relationships where one modality (e.g., Text) interacts with a pre-fused representation of two others (Vision + Audio).

Fig 2. Detail of explicit uni-modality, bi-modality, and tri-modality interactions.

3. Dual-Level Pooling

To handle the high dimensionality of these interactions, the authors use non-linear max-pooling at two stages:

• Word-level: To capture the most salient "moments" in the sequence.
• Modality-level: To distill the final representation for the final classification/regression layer.

Experiments & Results

The model was tested on the POM (Persuasive Opinion Multimedia) corpus, evaluating 12 traits (e.g., Confident, Passionate, Dominant).

SOTA Comparison

MEMI consistently outperformed strong baselines like MARN (Multi-attention Recurrent Network) and TFN (Tensor Fusion Network).

• Accuracy: For "Vividness," accuracy improved by nearly 10% absolute compared to MARN.
• Correlation: The Pearson correlation coefficient ($r$) showed massive improvements, particularly in "Expertise" and "Humorous," where previous models often struggled with near-zero or negative correlations.

Table 1. Performance across 12 traits using Accuracy, Mean Absolute Error (MAE), and Correlation (r).

Critical Insight: Why Does It Work?

The success of MEMI suggests that personality is a global trait, not a local one. Unlike speech recognition—where a sound must map to a specific letter at a specific time—personality is perceived through the accumulation and contrast of signals over time. By allowing the "seriousness" of a face at the start of a video to interact with the "punchline" at the end, MEMI captures the Inductive Bias necessary for social signal processing.

Conclusion & Future Work

MEMI provides a robust framework for multimodal interaction that prioritizes explicit cross-temporal relationships. While highly effective, the authors note that future versions should incorporate metadata (age, gender, location) to further refine these personality profiles. As AI moves closer to empathetic human-computer interaction, models that can "connect the dots" across time and modality will be essential.