Towards Increased Accessibility of Meme Images with Rich Face Emotion Captions

TL;DR

Internet memes are a pillar of modern digital communication, yet they remain a "black box" for visually impaired users. This paper presents a breakthrough by focusing on Reaction Memes. By moving beyond simple labels like "happy" or "sad," the authors developed a system that generates rich descriptions—like "a broad smile of mischievous delight"—using a novel Siamese ranking pretraining strategy.

Background: The Accessibility Gap in Meme Culture

While social media giants have deployed Automatic Alt-Text (AAT) systems, these are designed for natural photos. When a visually impaired user encounters a meme, an AAT might say "man in a yellow shirt," which completely misses the point of the meme. Since memes are "multi-modal constructions," understanding the facial emotion is the key to unlocking the humor.

The Challenge: Why Generic AI Fails

1. Irrelevant Details: Current SOTA captioners focus on objects/actions (e.g., "woman sitting on a sofa") rather than the subtle facial cues that drive the meme's narrative.
2. Subjectivity: Emotions are abstract. Two people might describe the same smirk differently, making it hard for models to learn from categorical labels alone.
3. Data Scarcity: No large-scale datasets existed for captioning emotions in the wild before this work.

Methodology: Stage-wise Emotional Intelligence

The authors propose a two-stage approach to bridge the gap between "seeing" a face and "interpreting" an emotion.

Stage 1: Siamese Ranking for Discriminative Features

Instead of asking a model to guess a "happiness score" (regression), which is noisy and subjective, the authors built a Siamese Network. The model looks at two faces and decides which one shows higher "Valence" (pleasantness) or "Arousal" (intensity). This "ranking" task forces the model to learn fine-grained visual differences.

Stage 2: Rich Caption Generation

The robust features from Stage 1 are passed to a Convolutional Captioning model. This model was trained on the authors' new Meme Face Emotion Captions Dataset (2,000 images, 6,000 captions).

Experimental Results & Human Validation

The model was tested against standard ImageNet-based captioners. The results showed a massive jump in "CIDEr" and "SPICE" metrics, which measure how "human-like" and "semantically rich" a caption is.

The "Real World" Test

The authors visited a foundation for the visually impaired. They found that providing Meme Text + Emotion Caption (e.g., "Meme Text: 'When you find $20 in your pocket' + Caption: 'A wide smile of pure joy and excitement'") allowed users to truly "get" the joke for the first time.

Critical Insight: The Value of Specialized "Small Data"

This paper proves that we don't always need massive datasets if we use Intelligent Pretraining. By using ranking on a large dataset (AffectNet) and fine-tuning captions on a small but high-quality dataset, the authors achieved SOTA results in a niche but high-impact domain.

Future Outlook

While this work masters "Reaction Memes," the next frontier is Contextual Awareness. Future models will need to understand who the character is (e.g., Leonardo DiCaprio) and the pop-culture context of the scene to provide a truly complete experience for visually impaired netizens.

Summary Takeaway: Accessibility isn't just about describing what's in a photo—it's about conveying the feeling and intent behind the pixels.