Can passenger EEG decode hazard perception in automated vehicles?

Passengers, not drivers, as the signal source

The paper asks whether the brain activity of a person riding in a highly automated vehicle can be decoded into two different safety-relevant judgments. Rather than treating the human as a driver who intervenes, it treats the human as a passenger whose neural responses supply an auxiliary supervision signal to the vehicle. The authors frame this as a way to support Safety of the Intended Functionality (SOTIF), the problem of automated systems encountering scenarios their training data did not cover.

The study separates two cognitive moments that are easy to conflate. Risk Prediction (RP) asks whether the current traffic interaction is likely to become risky during the approach phase, before anything hazardous has happened. Danger Identification (DI) asks whether an immediate hazard has actually been instantiated around or just after the event onset. RP is therefore an anticipatory question and DI a recognition question, and the paper keeps them as separate binary tasks with separate EEG time windows.

Why dangerous implies risky in the labeling scheme

The labeling design is the part that makes the two tasks comparable. Each ego–target interaction is described on two binary axes: Risky versus Not Risky, defined before onset as a target within 5 seconds of a possible hazardous maneuver, and Dangerous versus Not Dangerous, defined at or after onset as an active hazardous maneuver. The combination Not Risky but Dangerous is excluded by construction, leaving three valid levels: Safe, Low-Risk, and High-Risk.

This constraint is called Risk-aware Sequential Labeling (RSL). Its consequence is that RP is positive for both Low-Risk and High-Risk samples, while DI is positive only for High-Risk samples. In other words, DI is a stricter state nested inside RP, and every Dangerous sample is also counted as Risky for RP. Readers comparing the two accuracies should keep this asymmetry in mind: the two numbers come from different label definitions and different time windows, not from the same classification problem scored twice.

The time windows follow the same logic. RP uses only the pre-onset interval from −2.0 to −0.5 seconds, which keeps it strictly causal. DI uses −0.2 to 1.0 seconds, spanning onset and the immediate post-onset period.

A 3D convolutional recurrent network with two branches

The decoding model, a 3D Convolutional Recurrent Neural Network (3D-CRNN), takes multichannel EEG and reshapes it into an electrode-topology-informed 3D representation. Two branches process the RP window and the DI window separately, each extracting local spatial features, temporal features, and global features through convolutional stages. The branch features are concatenated along the temporal axis and passed to a gated recurrent unit with 32 hidden units, which models dependencies across the encoded sequences.

Training uses a weighted cross-entropy loss to counter class imbalance. The RP branch is optimized with RP labels and the DI branch with DI labels, and the fused representation is then optimized end-to-end with DI labels, so the final DI prediction draws on both branch-specific features and the fused temporal representation. The paper describes this combination of the Passenger Cognitive Model, RSL, and the 3D-CRNN decoder as the Passenger EEG Decoding Strategy.

Risk prediction is strong, danger identification is lower

The reported balanced accuracy (BA) is 95.3% ± 2.7% for RP. For single-subject DI, the 3D-CRNN reaches 85.0% ± 3.2% with RSL, up from 80.9% ± 3.9% without it. That gain is the paper's clearest internal comparison: adding the sequential labeling formulation improved DI for the proposed model and for most baselines in the single-subject setting.

The gap between 95.3% and 85.0% is not a sign that one task is simply harder in the same way. RP is scored on a pre-onset window with a broader positive class that includes both Low-Risk and High-Risk samples, while DI must isolate the High-Risk quadrant alone. The higher RP figure therefore reflects an easier separation boundary as well as model performance, and the two should not be read as a single ranking of decoding difficulty.

The advantage holds across event types

Event-wise analysis reports that the 3D-CRNN consistently outperforms the other models across different event types in both RP and DI. For DI with RSL, balanced accuracy is 77.8% in Automated Emergency Braking scenarios, 88.0% in cut-in scenarios, and 79.9% in pedestrian scenarios. The comparison set includes EGNet and ShallowConvNet at 80.3%, DeepConvNet at 77.6%, XGB-DIM at 73.4%, CTNet at 68.2%, and DSC-ConvLSTM at 72.2%.

These per-scenario numbers matter because they show the aggregate DI figure is not carried by one easy event category. Cut-in events produce the highest DI accuracy and emergency braking the lowest among the three reported types, so the scenario mix in a deployment would shift the expected performance.

Unseen subjects are where accuracy falls

Generalization experiments separate several kinds of variability. Cross-session DI reaches 77.0% ± 5.3% BA, which speaks to the same participants recorded in different sessions. Cross-subject evaluation reaches 77.4% ± 1.1% BA on seen subjects but 64.9% ± 8.5% BA on unseen subjects. The paper notes that the model achieves the highest Precision, Recall, and F1-score in cross-subject evaluation but the second-highest BA on unseen subjects.

The drop to 64.9% on unseen subjects is the most decision-relevant boundary in the paper. It means the decoder transfers only partially to new people, and the authors attribute this to individual differences in age, sex, cognitive style, risk tolerance, prior driving experience, and neural response patterns. Any claim that passenger EEG is ready as a real-time vehicle signal has to be read against this number rather than against the 95.3% RP figure.

A simulator proof of concept with 11 valid participants

The data come from a driving simulator study with 15 enrolled participants, of whom 11 provided valid data for analysis. Participants were treated as passengers, so driving experience was not a prerequisite. The authors describe the work as a controlled proof-of-concept study and list the moderate sample size, the simulator-based setting, and inter-subject variability as limitations.

Two further constraints shape how far the results travel. The RP and DI taxonomy covers observable ego–target interactions such as pedestrian crossing, vehicle cut-in, and front-vehicle emergency braking, so the authors present it as an operational taxonomy for interaction-based hazards rather than a complete account of all AV hazards. The paper also notes that fatigue, reduced vigilance, and habituation to repeated scenarios may have weakened EEG responses across sessions, which would affect both intra-subject and cross-session variability. The decoded outputs are described as auxiliary signals for downstream AV actions, and those actions are not studied here.

Pre-onset risk is the more transferable target

The paper's own numbers point to a specific priority rather than a general endorsement. RP reaches 95.3% ± 2.7% BA and is built on a strictly causal pre-onset window, while DI falls to 64.9% ± 8.5% on unseen subjects. If the goal is an auxiliary signal that works across people without per-user calibration, the pre-onset risk task is the stronger candidate in this evidence, and the danger-identification task is the one that most needs subject-independent modeling.

The scenario split reinforces the same reading. DI accuracy ranges from 77.8% in emergency braking to 88.0% in cut-in events, so the value of a passenger-derived danger signal depends on which interaction types dominate the operating environment. The paper's contribution is to show that passenger EEG carries both anticipatory and hazard-related information in a controlled setting; the practical threshold it sets is that cross-subject transfer, not within-subject accuracy, is the number to watch before treating this as a deployable safety input.

Sources used in this answer