Sentiment Analysis through Eye-Tracking: A Neural Network Approach to Job Interviews

TL;DR

Can your eyes reveal your personality before you even finish answering a question? This paper presents a novel, low-cost sentiment analysis model that uses Eye-Tracking and Artificial Neural Networks (ANN) to assist HR professionals. By tracking involuntary pupil movements through a standard webcam, the system identifies psychological patterns—like whether a candidate is remembering a fact or constructing an answer—to provide an objective layer of decision support in recruitment.

The Motivation: Moving Beyond Subjectivity

In the high-stakes environment of a job interview, human recruiters often rely on "gut feeling" or manual observation of body language. While experts are trained in this, the process remains inherently subjective. Historically, high-tech alternatives like polygraphs or functional MRI (fMRI) have been used to detect physiological responses, but these are either too expensive or easily manipulated by "tricks."

The authors identified a gap: the need for a non-invasive, low-cost, and scientifically grounded tool that can interpret the "windows to the soul"—the eyes—to help assess personality dimensions like Extraversion, Emotional Stability, and Openness.

Methodology: From Pupil to Personality

The system's architecture is a blend of computer vision and cognitive psychology. It follows a multi-step pipeline:

1. Input Capture: Using the WebGazer JavaScript library, the system tracks the retina's coordinates (x, y) and dwelling time via a standard webcam.
2. Normalization: Raw pixel data is normalized into a Cartesian plane relative to the eye region to ensure consistency across different screen resolutions and head positions.
3. Neural Network Classification: A Multilayer Perceptron (MLP) with two hidden layers processes these coordinates. The network is trained to classify eye movements into seven "Eye Accessing Cues":
   • Visual Remembered (VR) / Visual Created (VC)
   • Auditory Remembered (AR) / Auditory Created (AC)
   • Internal Dialogue (ID)
   • Kinesthetic Sensations (KI)
   • Visual Defocused (VD)

Model Architecture

The authors utilized a supervised learning approach because eye movement behavior is non-linear—different individuals linger on different coordinates for varying durations.

Figure: The Multilayer Perceptron architecture for classifying eye patterns.

Experiments & SOTA Results

The evaluation was conducted using the Eye-Chimera dataset (885 samples) for training and a live test with five volunteers. Volunteers were asked open questions based on the "Big Five" personality traits (e.g., "Tell me about a time you showed leadership" to measure Extraversion).

Key Findings:

• High Certainty: The model achieved high classification certainty in several categories. For instance, in identifying "Visual Remembered" patterns, the system reached a certainty score of 0.953.
• Behavioral Correlation: The data confirmed that specific personality questions triggered predictable involuntary eye movements. For example, questions regarding "Openness to Experience" frequently triggered "Internal Dialogue" (ID) or "Kinesthetic" (KI) patterns as candidates processed complex feelings.

Table: Relative frequencies of eye fixations across different volunteers and questions.

Critical Analysis & Conclusion

The core contribution of this work lies in its accessibility. By moving eye-tracking from specialized laboratories to a browser-based webcam setup, the authors have democratized a powerful psychological tool.

Limitations:

• Hardware Sensitivity: The system requires users to avoid glasses and maintain a steady head position, which might be restrictive in a natural interview setting.
• Dataset Diversity: While the Eye-Chimera dataset is a solid start, the small group of five volunteers for the final test suggests that more extensive cross-cultural validation is needed.

The Multi-modal Future: The most exciting takeaway is the potential for Decision Support Systems (DSS). Recruiters shouldn't be replaced by AI, but they can be empowered by it. By correlating eye-tracking data with existing HR metrics, organizations can achieve a more holistic and less biased understanding of their future employees.

Summary Takeaway: This work demonstrates that the subtle dance of a candidate's eyes contains rich, quantifiable data that can be decoded by relatively simple neural network architectures to provide a "second opinion" in human resource management.