Selective Perception: Boosting Emotion Recognition with Temporal-Band Attention

TL;DR

Most emotion recognition systems waste energy "watching" hours of neutral expressions to find seconds of joy. This paper introduces a multi-modal LSTM-RNN that uses Band Attention to prioritize specific brain wave frequencies (like the $\beta$ wave during stress) and Temporal Attention to physically skip redundant video frames. The result? SOTA accuracy while only looking at 10% of the data.

Problem & Motivation: The Redundancy Trap

In real-world Human-Computer Interaction (HCI), emotional data is "untrimmed." If you record someone watching a 2-minute funny clip, they might only laugh for 10 seconds.

1. Temporal Redundancy: Standard models (CNN-LSTMs) scan every single frame, leading to massive computational waste and "diluted" features where neutral expressions mask emotional peaks.
2. Frequency Blindness: EEG signals are often split into $\alpha ,\beta ,$ and $heta$ bands. However, existing models usually concatenate these features or treat them equally, ignoring the fact that certain bands become much more relevant depending on the emotional state (e.g., $\beta$ waves spiking during high mental stress).

Methodology: Two Attentions are Better Than One

The authors propose a sophisticated architecture comprising three subnetworks: Visual (VGG-16 based), EEG (topographical images), and a central Multi-layer LSTM-RNN fusion unit.

1. Band Attention (The "What" in EEG)

Instead of a simple fusion, the model uses the previous hidden state ($h_{n-1}$) to calculate weights for each EEG band.

• Insight: If the model detects the user is entering a high-arousal state, the Band Attention mechanism shifts its focus to the $\beta$ frequency, which is a known physiological marker for mental tension.

2. Temporal Attention (The "When" in Time)

This is the most aggressive efficiency booster. The model doesn't just process the "next" frame; it predicts the timestamp ($f_{n+1}$) of the next frame it wants to see.

• Reinforcement Learning: Because picking a frame is a non-differentiable "jump," the authors use Policy Gradient (Monte-Carlo) to train the agent to maximize classification accuracy while minimizing the number of steps.

Figure 1: The overall architecture showing the flow from raw signals to the attention-driven LSTM fusion.

Experiments & Results

The model was tested on the MAHNOB-HCI dataset, a gold standard for multi-modal affect analysis.

Visualization of "Jumping"

The visualization below shows the Temporal Attention in action. The model takes a few steps to "warm up" and then rapidly homes in on the "peak frames" where the subject's emotion is most visible, effectively ignoring the neutral "baseline" segments of the video.

Figure 2: Top bars show Band Attention weights; the bottom timeline shows the Temporal Attention skipping redundant frames.

Key Quantitative Trophies:

• Arousal Accuracy: 73.1%
• Valence Accuracy: 74.5%
• Efficiency: The model outperforms baselines while only requiring 10% of the emotional data input.

Critical Analysis & Conclusion

The genius of this work isn't just in the accuracy—it's in the biological intuition. By allowing the model to "glance" at the video like a human might (skipping the boring parts) and "tune" into specific brain frequencies, the authors achieve a more "organic" form of machine perception.

Limitations: Converting EEG to images is effective but loses the fine-grained temporal resolution of the raw signal. Future work might benefit from using Transformer-based architectures to replace the LSTM, allowing for even more complex long-range dependencies in the "peak frame" selection process.

Summary: This paper provides a blueprint for efficient, multi-modal systems that don't just process data—they curate it on the fly.