Can multimodal gaze, pupil, and head data improve auditory attention decoding?

A four-way choice instead of left versus right

Auditory attention decoding (AAD) asks which speaker a listener is following, using measurable signals rather than asking them. Most prior benchmarks reduce this to a binary choice between two speakers, often placed at the far left and right. MAESTRO instead asks the decoder to pick the attended talker among four simultaneous English speakers, so chance performance is 25%, not 50%.

That change matters because a two-way left/right decision can be made from coarse spatial cues alone. Choosing among four speakers at ±22.5° and ±67.5° from the listener's frontal axis requires distinguishing individual sources, not just sides of space. The paper argues this makes the task both harder and more informative about what a decoder is actually using.

The corpus itself covers 16 participants and 100 experimental trials, with 32-channel EEG, binocular gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data recorded at the same time. Four speech streams came from LibriSpeech and two background-noise streams from CHiME-Home, played through rear loudspeakers, with attended-speaker signal-to-noise ratios from 0 to 18 dB in a naturally reverberant room.

Gaze and head motion narrow the four-way choice

The paper's reasoning is that EEG carries the neural tracking of the attended speech envelope, while gaze, head orientation, and scene video indicate where the listener is oriented. Those behavioral signals cannot track a speech envelope well, but they can eliminate candidate speakers. In the baseline network, EEG embeddings are compared with each speaker's envelope by correlation over time, whereas gaze, IMU, and video embeddings are pooled and classified over the four speaker positions without any audio input. The two scores are added.

This design choice is visible in the recorded behavior. Gaze dispersion peaked at trial onset and offset and stayed stable in between, and head-movement magnitude peaked at onset and then stabilized from the third of six time bins onward. The authors read this as an initial orienting response followed by sustained attention and postural stillness during listening.

Behavioral signals were informative on their own. Without access to the speech, scene video reached 39.57%–44.69% under leave-one-subject-out evaluation, and gaze combined with video reached 44.61%–47.10%, both well above the 25% chance level. No behavioral combination matched EEG alone, but each added several points when combined with it.

Gains of 5.6 to 13 points over EEG alone

The headline comparison is between EEG-only decoding and configurations that add behavioral modalities. Within-subject, adding behavioral signals improved accuracy by 8.03 to 13.01 percentage points across window lengths. Under leave-one-subject-out (LOSO) evaluation, where the model is trained on 15 participants and tested on the held-out one, the gain was 5.58 to 10.77 points.

The two splits should not be compared cell by cell. Within-subject folds rotate through all 100 trial contents, while every LOSO fold is tested on the same held-out 20 trial contents, so the two halves of the results come from different test material. What holds in both is the direction: every winning combination contained EEG, and no purely behavioral combination was best at any window length.

EEG alone reached 43.63%–59.06% within-subject across window sizes, which the authors note is far below the 80%–99% reported for binary two-speaker decoding. They attribute part of that gap to task difficulty — four speakers, varying SNRs, background noise — and part to stricter evaluation that prevents data leakage. At the longest window, EEG alone still recovered the attended speaker at more than twice chance, and the full multimodal configuration reached 67.45% under LOSO.

The behavioral advantage grows as windows shrink

Decision windows are the stretches of signal a model uses for one prediction, tested at 5, 10, 15, 20, and 30 seconds. Window length controls how much temporal context is available. The multimodal benefit was largest where EEG had least context: under LOSO, the gain nearly doubled as the window shortened, from 5.58 points at 30 seconds to 10.77 points at 5 seconds.

The optimal combination also changed with window length rather than converging on one configuration. The authors interpret this as evidence that neural and behavioral signals are complementary, and they suggest practical systems would need adaptive, user- and context-dependent weighting rather than a fixed sensor set.

A permutation-style check supports the interpretation. The paper reports that adding behavioral modalities raised the contribution as well as the accuracy, and subject-level analyses showed each participant's best multimodal configuration beat EEG alone in most subject-window combinations, indicating the gains were not just an artifact of averaging across people.

An audio artifact the benchmark deliberately removes

MAESTRO's released audio contains a rendering artifact that could shortcut the task. The SNR adjustment scaled the attended channel before 16-bit encoding, clipping its peaks but not the competitors'. The attended envelopes therefore carry a lower crest factor, 11.1 dB against 20.5 dB, and a logistic classifier trained on eight summary statistics of a standardized envelope identified the attended speaker in 56% of windows against 25% chance.

Standardization does not remove this, because it fixes only mean and variance while skewness, kurtosis, sparsity, and dynamic range survive. The authors apply histogram equalization across the four envelopes in each window so all four share identical value distributions, which drops the probe to 26%. What remains is temporal ordering, the information cortical tracking depends on.

The transform is not lossless and may remove some genuine envelope information, but the authors state it can only reduce decoding performance, never inflate it, so the reported accuracies are conservative. The distortion still affects any downstream use of the raw audio, which the paper flags as a property of the released recordings.

Comprehension was high but gaze quality varied

Behavioral performance supports the usability of the recordings. Mean comprehension accuracy across all 16 participants was 83.2%, with all but one participant between 74% and 92%; Subject 1 scored 63%. The group mean rose from 80% in the first session to 87% in the second, suggesting a familiarization effect after the mandatory 10-minute break.

The option "I could not pay attention" was selected on only 1.4% of trials, and those trials were excluded from accuracy calculations. Correctness labels are released with the dataset so downstream researchers can include or exclude them.

Gaze data quality averaged 80% valid samples across trials and participants, with nine participants above 90%. Two participants, Subjects 8 and 12, had substantially lower validity at 27.3% and 42.4%, while the rest averaged 86.5%. Tracking loss came mainly from blinks, partial occlusions, and gaze outside the camera's range during natural movement. Pupil diameter was bilaterally symmetric and within the normal range, at 3.29 ± 0.62 mm left and 3.28 ± 0.57 mm right.

A fixed four-speaker scene limits transfer

The acoustic scene is fixed: the four loudspeakers occupy the same positions throughout, so orienting toward the attended speaker reduces to a choice among four known directions, and gaze dispersion stabilizes shortly after trial onset. The behavioral modalities therefore face a simpler problem than a realistic environment where sources move and their number is unknown. Repeating the paradigm across rooms with moving or repositioned sources would be needed to establish how far the multimodal benefit transfers.

The cohort is 16 native English speakers, comparable in size to established AAD datasets but small enough to limit cross-subject analyses and increase sensitivity to outliers such as Subject 1. The language restriction further narrows applicability.

The baseline also applies the same encoder design to every modality despite their differing temporal and noise characteristics, with only the EEG encoder receiving a spatial mixing layer. The authors present the results as a lower bound on achievable performance rather than an upper bound, and call for modality-specific encoders and fusion that weights modalities by signal quality.

EEG stays necessary while behavior sets the margin

The paper's most decision-relevant pattern is a division of labor rather than a replacement story. Every winning configuration contained EEG, and no behavioral combination was best at any window length, so the neural signal remains the core of the decoder. The behavioral streams set the margin, and that margin is largest exactly where EEG is weakest: short decision windows.

For anyone weighing sensor configurations, the practical threshold implied by these results is window length. At 30 seconds under LOSO, adding gaze, head motion, and video bought 5.58 points; at 5 seconds it bought 10.77. A system that can afford long integration windows gains less from extra sensors than one that must decide quickly.

The second implication is that no single multimodal configuration dominated across window lengths. That variability, together with differences across participants, is the paper's argument for adaptive modality weighting over a fixed sensor set — and the reason its benchmark reports all 15 input configurations rather than one headline number.

Sources used in this answer