AVEC 2019: Pushing the Frontiers of AI in Mental Health and Cross-Cultural Affect
AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition
The AVEC 2019 Challenge introduces three multimodal tasks: State-of-Mind (SoM) prediction, depression assessment with AI-driven virtual agents, and cross-cultural affect recognition. The paper defines standardized benchmarks using audio, visual, and deep learning features, establishing SOTA baselines using GRU and LSTM architectures to achieve competitive CCC and RMSE scores.
TL;DR
The 9th Audio/Visual Emotion Challenge (AVEC 2019) tackles three high-impact domains: tracking the "State-of-Mind," detecting depression using fully autonomous AI agents, and bridging the cultural gap in emotion recognition. Through the release of standardized corpora like USoM, Extended-DAIC, and SEWA, the organizers demonstrate that while deep learning can effectively model visual emotional universality, vocal biomarkers remain localized, and AI-driven clinical interviews present a significantly harder challenge than human-mediated ones.
Problem & Motivation: The Gap Between Lab and Life
Traditional affective computing often relies on "prototypical" data—actors performing emotions in controlled rooms. AVEC 2019 breaks this by focusing on naturalistic behavior.
The authors identify three critical pain points:
- State-of-Mind (SoM) Bias: Self-reports are subjective and rooted in past experiences (adaptation level). Predicting an "absolute" score is less meaningful than predicting the change relative to a baseline.
- The "Wizard-of-Oz" (WoZ) Crutch: In depression detection, previous successes relied on a human "wizard" controlling the virtual interviewer. When AI takes the reins, the system's ability to maintain rapport and elicit diagnostic cues drops significantly.
- Language Barriers: Acoustic features (prosody, pitch) that work for European languages often fail for tonal languages like Chinese, where pitch carries grammatical meaning rather than just emotional weight.
Methodology: Deep Representations and Contextual Modeling
The challenge utilizes a robust pipeline of expert-knowledge features and deep learning representations.
Architecture Analysis
- Visual: Uses OpenFace for Facial Action Units (FAUs) and deep features from ResNet-50 and VGG-16 pre-trained on the Aff-wild dataset.
- Audio: Employs eGeMAPS (standard acoustic parameters) and Deep Spectrum features where mel-spectrograms are treated as images and passed through CNNs (DenseNet).
- Sequence Modeling: All tasks utilize Gated Recurrent Units (GRU) or Long Short-Term Memory (LSTM) networks to capture the temporal dynamics of emotion.
Note: Table 1-3 summarizes the vast multimodal datasets used to train these complex temporal models.
Experiments & Results: The Reality of Autonomous AI
The baseline results offer a sobering look at the complexity of AI in healthcare.
- Depression Detection: The performance on the Extended-DAIC corpus (AI-driven) was lower than previous human-driven datasets. This suggests that AI agents do not yet "interview" as effectively as humans, making it harder to spot depression indicators.
- Cross-Cultural Universality: Visual features (FAUs) proved significantly more transferable across German, Hungarian, and Chinese cultures than audio. In Mandarin, the pitch contour is so tied to linguistics that the "emotional" signal becomes obscured.
- Emotional Inertia: For the State-of-Mind task, the best results came from a "mixed" training strategy—training on static scores but evaluating on the dynamic change (CAi - CA0). This aligns with psychological theories that human emotion is nested and relative.
The results show a clear advantage for deep visual representations (ResNet) in depression detection over traditional acoustic descriptors.
Critical Analysis & Conclusion
Takeaway
AVEC 2019 reinforces that context is king. Whether it is the cultural context of a tonal language or the interpersonal context of a clinical interview, AI must adapt. The successful use of Curriculum Learning (presenting easier examples first) and Dynamic Evaluation marks a shift from simple regression to modeling human-like "emotional inertia."
Limitations & Future Work
The primary limitation identified is the "interaction gap" in AI-driven interviews. Future research should not just focus on detecting signals but on generating agent behaviors that actively encourage the user to reveal their emotional state. Furthermore, domain adaptation techniques are urgently needed to refine how audio features are processed in Cantonese and Mandarin within affective computing frameworks.
AVEC 2019 serves as a benchmark for the next generation of AI: one that is culturally aware and clinically capable.