Beyond Self-Reports: Fusing ECG, EDA, and fNIRS for a Holistic View of Human Emotion

TL;DR

Quantifying human emotion in Human-Computer Interaction (HCI) is notoriously difficult due to the "noise" of cognitive effort. This paper introduces a multimodal approach that fuses ECG/EDA (autonomic signs) with fNIRS (cognitive signs). By weighting these data streams, researchers achieved a remarkable 93.93% accuracy in classifying emotional states, proving that our brains and hearts provide a more complete picture when analyzed together than in isolation.

The Confounding Trap: Emotion vs. Task Load

In the world of affective computing, heart rate is a double-edged sword. While it correlates with arousal (the intensity of an emotion), it is also heavily influenced by attention and cognitive effort. If a user is concentrating hard on a Tetris game, their heart rate might increase—is that because they are excited (emotion) or simply thinking hard (workload)?

This "confounding effect" has historically limited the accuracy of physiological emotion sensors. Prior work often struggled to separate the sympathetic activation triggered by stress from that triggered by interest.

Methodology: The Multimodal Fusion

The researchers utilized a three-pronged sensor approach to map emotions onto Russell’s Circumplex Model (Arousal vs. Valence):

1. ECG (Electrocardiogram): Beyond simple heart rate, they analyzed the PQRST waveform (including R-wave height and QT intervals) to find more sensitive markers of emotional shift.
2. EDA (Electrodermal Activity): Used as a primary index for arousal based on skin conductance (sweat gland activity).
3. fNIRS (Functional Near-Infrared Spectroscopy): Measures oxygenated hemoglobin changes in the prefrontal cortex to capture the cognitive workload.

System Architecture and Feature Extraction

The team extracted 44 features per subject, including the mean, standard deviation, and extrema of intervals like RR, QRS, and ST.

Figure: The PQRST waveform analysis used to extract granular physiological features.

The breakthrough came in the Fusion Stage. By assigning weights to the predictions of the ECG/EDA classifier and the fNIRS classifier, the system could "verify" an emotional state. If the heart suggests high arousal but the brain shows low workload, the confidence in a purely emotional reaction increases.

Experimental Setup: From Music Videos to Tetris

To ensure a wide range of emotional responses, the study used three stimuli types:

• DEAP Dataset: Music videos validated for high/low valence and arousal.
• MATB (Multi-Attribute Task Battery): A cockpit-style simulation to induce varying workload levels.
• Tetris: Adjusted for difficulty (speed) to trigger both frustration and flow states.

Results: The Power of Weighted Fusion

The results confirm that the "Heart + Brain" approach significantly outperforms the "Heart Only" model.

• ECG Alone: Achieved peak average accuracy around 70% for valence using SVM.
• Fused Model (ECG + fNIRS): When predictions were combined via Naive Bayes with higher weight assigned to ECG, accuracy surged to 93.93%.

Table: Accuracy Comparison across subjects for Valence classification.

Critical Insight: Why Does Fusion Work?

The core insight is that autonomic nervous system (ECG/EDA) data is highly reactive but ambiguous. By "complementing" it with fNIRS cognitive data, the machine learning model gains a context-aware filter. It effectively asks: "Is this heart rate spike explained by the brain's workload?" If not, it is likely a pure emotional response.

Conclusion and Future Outlook

This study serves as a proof-of-concept for high-accuracy affective computing in interactive environments. While the sample size (9 subjects) is small, the methodology provides a roadmap for:

• Adaptive Systems: Games or training simulators that adjust difficulty based on real-time emotional frustration vs. cognitive engagement.
• Health Monitoring: More accurate stress detection that doesn't trigger "false positives" during productive work.

Limitations: The current model uses a simple linear weighting for fusion. Future research should explore Deep Neural Networks to learn non-linear relationships between hemodynamic (brain) and electrical (heart) signals.

Takeaway: To truly understand how a human feels, we must look at both the "logic" of the brain and the "passion" of the heart.