APSE: Bridging the Affective Gap with Hierarchical Attention and Polarity-Sensitive Embeddings

Executive Summary

TL;DR: The APSE (Attention-aware Polarity-Sensitive Embedding) network redefines Emotion-Based Image Retrieval (EBIR) by explicitly modeling the hierarchy of human emotions. By integrating spatial attention at different layers of a CNN and introducing a polarity-sensitive loss function that generates adaptive hard negatives, it achieves a massive performance leap (up to 10% in mAP) over previous benchmarks.

Positioning: This work is a significant advancement in Affective Computing. It moves beyond simple classification to a more nuanced metric learning approach, treating emotions not just as discrete labels but as a hierarchical structure of human perception.

The "Affective Gap" Problem

In traditional Content-Based Image Retrieval (CBIR), we look for visual similarity (e.g., "find more dogs"). EBIR is harder because it targets abstract semantics. Previous SOTA suffered from two main flaws:

1. Context Blindness: They treated all pixels equally, missing the "informative regions" (like a facial expression or a specific color contrast) that actually trigger the emotion.
2. Polarity Negligence: They often failed to distinguish between positive and negative polarities, occasionally retrieving a "sad" image for a "happy" query because of shared low-level textures.

Methodology: The APSE Framework

APSE solves these issues through a dual-pronged architecture:

1. Hierarchical Attention Mechanism

Humans perceive emotions at different levels of abstraction.

• Shallow Layers: Focus on "Polarity" (Positive/Negative), often triggered by low-level cues like color and texture.
• Deep Layers: Focus on "Specific Emotions" (Amused, Sad, Awe), requiring high-level semantic understanding.

These attended features are fused using Cross-Level Bilinear (CLB) pooling, which captures the interaction between these levels rather than just concatenating them.

Fig 1: The Attention Map Generation process. Note how different layers are supervised by different labels (Polarity vs. Category).

2. Polarity-Sensitive GEP Loss

To ensure the embedding space is "well-organized," the authors proposed the Generated Emotion-Pair (GEP) Loss.

• Inter-polarity constraints: Ensure "Happy" is far from "Sad."
• Intra-polarity constraints: Ensure "Amusement" is distinguishable from "Contentment," even though both are positive.
• Adaptive Negative Generation: Using the confidence scores from the attention module, the system creates "Harder" negative examples in the feature space to push the model to learn tougher boundaries.

Experiments and Results

The model was tested against 20+ baselines across four datasets (FI, IAPSa, ArtPhoto, Abstract).

Key Results (FI Dataset):

• mAP8 (Specific Emotion): Increased from 0.6395 (SOTA) to 0.7433.
• mAP2 (Polarity): Reached 0.9030.
• Visual Evidence: APSE successfully identifies the emotional "trigger" in images, such as the facial regions in a statue or specific textures in an abstract painting.

Table 1: APSE outperforms both traditional descriptors and deep metric learning baselines.

Deep Insight: Why it Works

The brilliance of APSE lies in its biological/psychological intuition. By supervising shallow layers with binary polarity and deep layers with category labels, the network mimics the human "fast and slow" emotional response. The GEP loss then acts as a sophisticated "tuner," ensuring that the geometry of the latent space reflects the real-world subjective similarity of emotions.

Conclusion and Future Outlook

APSE represents a major milestone in making AI "emotion-aware." While it excels at retrieval, it still faces challenges with emotional ambiguity (where one image evokes multiple feelings).

Future Work: The authors suggest moving toward continuous label spaces (like Valence-Arousal) and exploring Label Distribution Learning to handle the inherent subjectivity of human emotions even more effectively.