Decoding Player Emotions: A Multimodal Approach to UX Evaluation in Serious Games

TL;DR

Understanding how a player feels during an educational game is shifting from subjective surveys to objective biosignals. This paper presents a 2-stage machine learning framework that fuses EEG (neurological) and Heart Rate (physiological) data to classify emotional states with over 91% precision, providing a roadmap for real-time, data-driven User Experience (UX) design.

The "Subjectivity" Problem in Serious Games

Serious games—games designed for education or training—rely heavily on player engagement. However, traditional UX evaluation is stuck in the past, predominantly using post-session questionnaires. These are limited by:

• Memory Bias: Players forget specific moments of frustration or excitement.
• Snapshot Limitations: Surveys provide a summary, not a real-time "emotional map."

The authors argue that for serious games like "Programming Hero," objective measures are necessary to capture the nuanced emotional transitions between being "Focused" (Activated) and "Bored" (Deactivated).

Methodology: The 2-Stage Fusion Architecture

The core innovation lies in how the authors treat the relationship between the brain and the body. Instead of simple data concatenation, they use a hierarchical inference model.

1. Data Acquisition

The study used affordable, wearable tech:

• Neurosky Mindwave Mobile 2: Captured Alpha and Beta brainwaves (Neurological).
• Mi Smart Band 4: Captured heart rate via PPG (Physiological).

2. The 2-Stage Learning Framework

• Stage 1 (Prediction): A Decision Tree model predicts physiological metrics from neurological inputs. This step essentially "aligns" the two different data streams.
• Stage 2 (Classification): An XGBoost learner takes the features (Alpha High/Low, Beta High/Low) plus the predicted heart rate to categorize the user as Activated (Anxious, Focused, Stressed) or Deactivated (Relaxed, Bored).

Figure 1: The architecture of the proposed machine learning approach for emotion classification.

Experimental Insights & Results

The study involved computer science students playing a programming game. The analysis revealed that Heart Rate (Beats-per-15s) was actually the most critical feature in predicting the emotional state, even more so than individual brainwave bands.

Key performance metrics:

• Algorithm Superiority: Ensemble methods (XGBoost, AdaBoost) significantly outperformed traditional models like SVM or KNN.
• XGBoost Accuracy: Achieved a Recall of 0.920, proving that the model is highly sensitive to shifts in the player's emotional state.

Figure 2: Accuracy, Precision, and Recall comparison across different ML techniques. XGBoost emerges as the clear winner.

Why This Matters: From Evaluation to Adaptation

The implications of this work extend beyond just "measuring" UX. If a game can detect in real-time that a player's state is "Deactivated" (Low Arousal), it could:

1. Dynamic Difficulty Adjustment: Pivot to a more challenging "test" mode.
2. Guidance Injection: Offer a hint if the player is in a "Stressed but Low Attention" state.

Limitations & Future Outlook

While the results are promising, the study used a small cohort (12 participants). As the field moves toward larger datasets, the integration of Deep Learning (like RNNs or LSTMs for time-series biosignals) could further refine the temporal resolution of these emotional maps.

Conclusion

This research successfully bridges the gap between physiological signals and UX evaluation. By proving that ensemble machine learning can accurately interpret "the body's response" to code-learning, it sets the stage for a new generation of emotion-aware educational software.