F-Score Fusion: Bridging Audio and Visual Cues for Robust Emotion Recognition

TL;DR

Recognizing human emotions is complex because cues are often split between what we say and how we look. This paper by Gera and Bhattacharya presents an automatic bi-modal framework that fuses audio features (MFCC, Pitch, Energy) and visual geometric features. By using a novel F-score based fusion strategy, they outperformed existing benchmarks by 9% on the eNTERFACE dataset.

Background: Beyond Uni-modal Limits

Historically, emotion recognition focused on either facial expressions (Video) or prosody (Audio). However, humans are multi-modal. A person might keep a "stiff upper lip" while their voice trembles with fear. The authors argue that while Video excels at identifying high-movement emotions like Surprise, Audio is much more reliable for "internalized" emotions like Sadness.

The Synchronization Challenge

Most prior works used staged datasets where actors didn't speak while expressing emotions. Real-world interaction involves speech, where lip movements (visemes) often obscure emotional facial cues. This paper tackles synchronized data where speech and expression happen simultaneously.

Methodology: The "Smart" Fusion Choice

The core innovation lies in how the system decides which modality to trust. Instead of a simple "majority vote," the authors use an F-score Decision Matrix.

1. Feature Extraction

• Visual: The system tracks 66 facial points automatically using a subject-independent aligner. It extracts 17 geometric features (distances/angles) robust to head rotation and scale.
• Audio: It captures both frame-level temporal features (41-dim) and global signal-level statistics (60-dim), including MFCCs and energy ratios of voiced/unvoiced segments.

Figure 1: Automated tracking of 66 facial landmarks used for geometric feature extraction.

2. Multi-Classifier Setup

The authors utilize four base models:

• Video HMM (Temporal changes) & Video SVM (Peak frame).
• Audio HMM (Prosodic flow) & Audio SVM (Signal statistics).

3. F-score Decision Logic

If Classifier A predicts "Anger" and Classifier B predicts "Disgust," the system checks which classifier has historically been more "reliable" for those specific classes using the F-score computed during cross-validation. This allows the model to dynamically follow the "expert" modality for each emotion.

Experiments and Results

The study demonstrates a clear complementarity between modalities. Audio classifiers were significantly better at detecting Anger (63% accuracy) and Sadness (73%), whereas the Video HMM excelled at Happiness (61%).

The Fusion Leap

By combining all four classifiers via F-score fusion, the accuracy rose to 54.28%, significantly higher than the 45.23% achieved by previous state-of-the-art methods.

Table: Comparison of various fusion combinations. Note how fusing all four models provides the highest overall accuracy.

Critical Insight: Why it Works

The authors measured the Correlation Index (${\rho }_n$) and Q-statistic. They found that within the same modality (e.g., Audio SVM and Audio HMM), the correlation is high ($\rho \approx 0.39$), meaning they make the same mistakes. However, between Audio and Video, the correlation is nearly zero. F-score fusion thrives on this independence; if the video is confused by lip movements, the audio prosody provides the ground truth.

Conclusion and Limitations

While highly effective for its time, the 54.28% accuracy indicates that emotion recognition in-the-wild remains a "hard" AI problem. The current approach relies on handcrafted geometric features; modern iterations would likely replace these with Deep Neural Networks (CNNs/Transformers). However, the logic of class-specific fusion remains a powerful takeaway for any multi-modal ensemble system today.

Takeaway: Don't just average your models; find out which model is an expert in which category and let it lead the decision.