Stimuli-Aware VEA: Bridging the Affective Gap with Psychological Intuition

TL;DR

Recognizing emotions in images is far more complex than identifying a "dog" or a "cat." This paper argues that emotions are triggered by specific stimuli—colors, object interactions, and facial expressions. By mimicking the psychological S-O-R (Stimuli-Organism-Response) model, the authors developed a multi-branch network that outperforms existing SOTA methods on major benchmarks through a novel hierarchical loss function.

The "Affective Gap": Why Traditional CNNs Struggle

In traditional computer vision, we map pixels to semantics (e.g., "this is a car"). However, Visual Emotion Analysis (VEA) requires mapping pixels to feelings. This is known as the Affective Gap.

Previous deep learning approaches simply fed the whole image into a backbone like ResNet and hoped the model would "figure out" the emotion. The authors found this insufficient because:

1. Global Context Matters: A sunset evokes sadness or awe through color and lighting, not just objects.
2. Semantic Correlations: A "white rose" means "joy" at a wedding but "sadness" at a funeral.
3. Human Empathy: Facial expressions in an image directly infect the viewer’s emotional state.

Methodology: The S-O-R Architecture

The proposed framework is divided into three distinct stages that mirror human cognitive processes.

1. Stimuli Selection (S)

Instead of viewing the image as a flat grid of pixels, the model identifies three key stimuli:

• Global: The entire image (focusing on color/texture).
• Semantic: Top-N objects detected via Faster R-CNN.
• Expression: Detected and aligned faces using Dlib.

2. Feature Extraction (O - Organism)

The "Organism" phase processes these stimuli using specialized sub-networks:

• Global-Net: A ResNet-50 capturing the overall "vibe."
• Semantic-Net: A sophisticated dual-LSTM setup (Attention LSTM + Correlation LSTM) that mines how different objects in a scene relate to one another.
• Expression-Net: A ResNet-18 pre-trained on facial expression data (FER2013) to capture "empathy."

3. Emotion Prediction (R - Response)

The final stage isn't just a simple classifier. The authors introduce a Hierarchical Cross-Entropy Loss.

Using Mikel's Wheel, they group the 8 emotion categories into two polarities: Positive (Amusement, Awe, etc.) and Negative (Sad, Anger, etc.). The loss function penalizes the model more if it confuses a "Positive" emotion for a "Negative" one (a "Hard False" error) compared to confusing two emotions within the same polarity.

Experimental Results & Insights

The model was pitted against SOTA methods like WSCNet and MldrNet across four datasets (FI, EmotionROI, ArtPhoto, IAPSa).

• SOTA Performance: On the large-scale FI dataset, the model reached 72.42% accuracy, a significant jump over the baseline ResNet-50 (67.53%).
• The Power of Color: Disabling color channels (using only Y-channel) caused accuracy to drop significantly, proving color is a primary stimulus.
• Object Relationships: The dual-LSTM Semantic-Net proved superior to simple fully connected layers, confirming that the relationship between objects is more emotional than the objects themselves.

Deep Insight: Beyond Just Accuracy

What makes this work stand out is its interpretability. By visualizing the Class Activation Maps (CAM) and attention weights, we can see why the model made a choice. For instance, in an "Excitement" image, the Semantic-Net attends to both "happy people" and "a boat in a drift," capturing the adrenaline of the scene.

However, the paper honest acknowledges failures. In complex cases—like a "sad clown" where a smile masks sad eyes—the model still struggles. This suggests that the next frontier in VEA isn't just better vision, but incorporating common sense and high-level reasoning.

Conclusion

This stimuli-aware approach demonstrates that when we treat AI architecture as a reflection of human psychology rather than just a mathematical optimization problem, we get models that are not only more accurate but also more aligned with how we perceive the world.