Speech Emotion Recognition: Bridging Traditional ML and CNNs

TL;DR

This research investigates the effectiveness of two distinct paradigms for Speech Emotion Recognition (SER) using the RAVDESS dataset. By comparing traditional classifiers like SVM and Random Forest (using MFCCs) against a Convolutional Neural Network (using Mel-spectrogram images), the study reveals that while Deep Learning is promising, classic SVMs still hold a "Polynomial" edge in accuracy for specific emotion classifications.

Problem & Motivation

Human speech is inherently non-stationary; the frequencies we produce change rapidly due to random variations in the vocal tract. Recognizing emotion automatically is significantly harder than speech-to-text because emotional cues are often buried in subtle "quefrency" shifts rather than just linguistic content.

Existing research often struggles with:

• Dataset Scarcity: High-quality emotional audio (like RAVDESS) is expensive to record.
• Feature Engineering: Finding the right balance between frequency resolution and computational cost.
• Overfitting: Deep models with millions of parameters often memorize the actors' voices rather than the emotions themselves.

Methodology - The Core

1. Traditional Feature Extraction: MFCCs

The authors utilize Mel-Frequency Cepstral Coefficients (MFCCs). This process mimics human hearing by applying a Mel-scale—which is more sensitive to changes in low frequencies than high frequencies.

• Windowing: Using Hamming windows to maintain signal continuity.
• Cepstrum Calculation: Applying a Discrete Cosine Transform (DCT) to the log of the spectrum to isolate the "vocal tract" components from the "excitation" source.

2. Deep Learning: Spectrograms as Images

For the CNN approach, the audio is converted into Mel-spectrograms—2D visual representations of the audio. These images (128x128) are fed into a modified LeNet-5 architecture.

The rationale? CNNs are world-class at identifying spatial patterns, and emotional prosody often manifests as specific visual patterns (slopes, intensities) in a spectrogram.

Experiments & Results

Conventional Machine Learning

The team tested kNN, SVM, Random Forest, and MLP. The SVM with a Polynomial Kernel was the clear winner.

• Performance: 70.3% accuracy on 6 emotions.
• Complexity vs. Accuracy: Increasing the number of MFCC coefficients from 13 to 27 improved accuracy but doubled the training time.

The CNN Challenge

The CNN achieved a validation accuracy of 68.7%. interestingly, the authors applied SpecAugment (masking blocks of time or frequency). While this successfully stopped the model from overfitting (closing the 22% gap between training and validation), it actually lowered the total accuracy. This suggests that in SER, even "empty" or "noisy" parts of the spectrum might contain vital emotional information that shouldn't be masked.

Confusion Matrix Analysis

The error analysis reveals deep insights into human expression:

• Easy to Identify: Neutral and Calm (90%+ accuracy).
• Complex Confusion: Anger is frequently confused with Disgust, and Surprise is often mistaken for Happiness. This aligns with psychological theories where these emotions share similar arousal levels.

Critical Analysis & Conclusion

Takeaway

The study proves that for researchers with limited computational resources, a Polynomial SVM using 27 MFCCs is a robust, lightweight, and highly accurate solution for SER.

Limitations

The CNN used was relatively shallow (LeNet-5). While this prevented some overfitting, it may lack the expressive power of modern ResNets or Transformers (Attention mechanisms) which could better capture temporal dependencies in speech.

Future Work

The authors point toward Convolutional Recurrent Neural Networks (CRNNs) as the next step—combining the spatial feature extraction of CNNs with the temporal sequence modeling of LSTMs to capture how an emotion evolves over the course of a sentence.