Precise Mouth Geometry: A Gradient-Based Approach to Emotion Analysis

TL;DR

This research introduces a robust, geometric-driven method for extracting mouth landmarks using vertical gradient analysis and morphological operations. Achieving over 95% accuracy in horizontal feature detection, it provides a lightweight yet effective framework for interpreting human emotions without the heavy computational overhead of holistic appearance-based models.

Background & Motivation: Moving Beyond Coarse Detection

In the realm of Human-Computer Interaction (HCI), facial expressions are the most direct window into user sentiment. Historically, researchers have leaned on two pillars: Appearance-based methods (analyzing raw pixel intensities) and Geometric-based methods (measuring the spatial relationship of landmarks).

While modern trends favor deep learning, the authors argue that geometric methods are undervalued. Coarse detectors like Viola-Jones are excellent at finding where a mouth is, but they lack the precision needed to define the shape of a smile or a pout. The motivation here is to bridge that gap—providing a simple, mathematically sound way to extract exact mouth contours that are resilient to illumination changes.

Methodology: The Power of Vertical Gradients

The core insight of this paper is that vertical intensity changes are significantly more informative for lip segmentation than horizontal ones. The process follows a strict 4-step pipeline:

1. Vertical Gradient Calculation: Instead of a full 2D gradient, the authors use a 3x3 vertical derivative operator to isolate horizontal edges (the lips).
2. Normalization & Squaring: To suppress noise and highlight the lip edges, the gradient values are squared and adjusted via MIN-MAX normalization.
3. Kernel Filtering & Thresholding: Specific matrices are used to sharpen the edges. A threshold (best at 250) is applied to create a binary mask of the mouth.
4. Morphological Closing: To handle "broken" edges where a contour might split, a "closing" operation (dilation followed by erosion) is used to unify the mouth shape.

Fig 1. Gradient distribution for different emotions: (a) Happy, (b) Neutral, (c) Sad.

Feature Extraction Logic

Once the shape is isolated, the method extracts six key points:

• $c_L,c_R$: The leftmost and rightmost extremes.
• $c_U,c_B$: The topmost and bottommost points.
• $W,H$: Derived Width and Height.

Experimental Results: High Precision in Horizontal Planes

The method was validated using the Yale Face Database. The results indicate a specialized proficiency in horizontal tracking:

Fig 2. Visualization of the extracted geometric corners and the calculated mouth width/height.

The "Upper Lip" Challenge

One critical finding was the difficulty in detecting the upper lip ($c_U$), which only reached 43.59% accuracy. This is attributed to the subtle gradient transition between the nose and the Philtrum, which often confuses simple gradient-based thresholds.

Deep Insight & Conclusion

The strength of this work lies in its Inductive Bias: it assumes that the most important information for mouth-based emotion lies in the horizontal stretch and the bottom-lip curve. By ignoring redundant horizontal gradients, the method gains speed and robustness against shadows that typically fall vertically across the face.

Takeaway for Practitioners: While deep learning models (like MediaPipe or Dlib) are industry standards, this gradient-based approach offers a "white-box" alternative. For systems with extremely limited power (IoT devices, embedded sensors), these geometric features provide a high-accuracy signal (Width/Right Corner) that can be fed into a tiny SVM or Random Forest classifier for efficient emotion detection.

Future Work: Integrating these mouth features with similar eye-region trackers will likely mitigate the lower accuracy of the upper-lip detection, creating a holistic facial geometry suite.