Can agentic LLM architecture search improve EEG brain-computer interface decoding?

One architecture must serve three task families

The paper's target is not the best model for a single EEG task. It is a single architecture that performs well across motor imagery, emotion recognition, and sleep staging, three families that differ in channel layout, temporal scale, and label space. That choice shapes everything else: each candidate is trained and validated separately on each task family, and the combined validation results decide which candidates become parents and which becomes the final champion.

The 14 datasets are pooled within each family, so one model instance is trained per task. Motor imagery and emotion recordings are mapped to a 65-channel template with 800 samples at 200 Hz, while the sparser sleep recordings use an eight-channel, 3,000-sample representation. Validation and test scores are still reported per dataset, with equal dataset weights inside a task and equal task weights across tasks.

This aggregation matters for reading the headline number. The 64.16% is an average test balanced accuracy over tasks and datasets under that weighting, not a score on any single benchmark. Balanced accuracy here is the mean recall over the classes present in each split, which keeps a majority class from dominating the score.

A persistent candidate pool drives refinement

The Designer Agent runs Pool-Guided Architecture Discovery. Each search has six rounds of eight candidates, evaluated across the three task families, for 48 proposals and up to 144 independent task-training runs. Round one is eight fresh proposals. Later rounds keep two parents, give each three refinements, and add two fresh proposals, so the batch stays at eight.

Parents are chosen from a persistent pool ranked by validation results, and they hand their architecture code plus validation learning curves for all three tasks to the next round. Refinements can therefore target observed strengths and weaknesses, while the two fresh proposals keep exploring outside the current parents. Every candidate must pass validity and training-compatibility checks, and only architectures that complete all three tasks are eligible for parent or champion selection.

The constraints are strict: fewer than one million trainable parameters in every task-specific instantiation, no recurrent architectures, and training from scratch under a fixed protocol of 40 epochs with AdamW, batch size 512, and BF16 precision. Six LLMs ran searches through their command-line interfaces, including Claude Opus 5.5 and GPT-5.6 Sol, producing 269 architectures trained on all three task families.

Early curves predict which candidates are worth finishing

Because every proposal is evaluated on three task families, screening is where the cost lives. The Forecaster Agent performs Performance Estimation from Early Knowledge, or PEEK. It receives only the architecture code, the training protocol, and early learning curves, and predicts the full-budget validation score that a candidate would reach after 40 epochs.

The retrospective test is direct: using ten observed epochs, PEEK reduced mean absolute error in predicting average validation balanced accuracy from 2.20 to 1.36 percentage points, a 38.1% reduction relative to the best-observed-score baseline. In other words, the forecast error fell by roughly 0.84 percentage points of balanced accuracy when only the first quarter of training was visible.

The selection payoff is separate and more mixed. Retaining three candidates per round preserved a full-budget winner in 91.7% of rounds, with an estimated 44.9% reduction in training epochs. But the paper also reports that PEEK's top-two selection retained fewer combined-score winners than the best-observed-score baseline. Forecast accuracy and winner retention are different measures, and the second is where the advantage narrows.

The champion beats REVE by 0.29 points on bAcc

The strongest discovered architecture, from the Claude Opus 5.5 search, reached 64.16% average test balanced accuracy against 63.87% for REVE, the strongest of the ten baselines on this metric. The comparison set is six conventional EEG models trained from scratch — EEGNet, DeepConvNet, TSception, EEG-Conformer, AttnSleep, and FAST — plus four foundation models fine-tuned from pretrained checkpoints: LaBraM, CBraMod, LEAF, and REVE.

The margin is small enough that the choice of metric decides the ranking. The paper states that REVE leads on weighted F1 and Cohen's Kappa, the other two reported test measures, which use the same averaging as balanced accuracy. So the honest summary is a split decision: AutoBCI's champion wins on balanced accuracy, the metric the headline emphasizes, while the strongest foundation-model baseline wins on the other two.

All baselines used the same task partitions and validation-based checkpoint selection as AutoBCI, and test data stayed out of search and selection. That makes the comparison internally consistent, but it also means the result is a statement about this protocol and these ten baselines, not about foundation models in general.

Search improves every run, but the ceiling is close

Across the six LLM-guided searches, every search improved its own best validation score, which is the paper's evidence that the generate-evaluate-refine loop is doing work rather than producing noise. The variation across LLMs is part of the result: the champion came from one model, and the paper reports the six searches as separate runs rather than pooling them into a single ensemble.

The more decision-relevant comparison is between the discovered architecture and the pretrained foundation models. A search constrained to under one million parameters and trained from scratch on pooled task data edged past a fine-tuned foundation model on balanced accuracy by 0.29 points. That suggests architecture discovery can compete at this scale, but it does not show that discovered architectures dominate, since the same baseline leads on two of three reported metrics.

The paper's own framing supports this reading: it presents the champion as exceeding the strongest baseline by 0.29 percentage points, and separately notes that REVE leads on weighted F1 and Kappa. Both statements belong together when judging whether the search was worth its cost.

Use PEEK to cut training, not to pick the winner

The two PEEK results point in different directions, and the distinction is practical. If the goal is to estimate how a candidate will finish, ten epochs of evidence cut prediction error by 38.1%, from 2.20 to 1.36 percentage points of balanced accuracy. If the goal is to keep the eventual best candidate alive, retaining three per round preserved a full-budget winner in 91.7% of rounds while cutting an estimated 44.9% of training epochs.

But top-two retention underperformed the best-observed-score baseline on combined-score winners. A reader deciding how aggressively to prune should therefore treat the retention count as the lever: three candidates per round is the setting the paper reports as preserving winners, while tighter pruning is where the forecast's ranking quality becomes the binding constraint.

These numbers come from retrospective evaluation on completed 40-epoch runs, where full-run outcomes supply the ground truth. They describe how well early curves predict within this search setup, not how a forecaster would behave on a different task family or training budget.

A narrow win under a fixed protocol

Several features of the setup bound what the result supports. The search space excludes recurrent architectures and caps every task-specific model below one million trainable parameters, so the discovered champion is the best of what this constrained space and this agent loop produced, not the best possible EEG decoder. Candidates are trained from scratch for 40 epochs, while the foundation-model baselines are fine-tuned from pretrained checkpoints.

The data are also shaped by the pooling and subsampling choices. Sleep staging retains 25%, 40%, and 50% of examples in Sleep-EDF, HMC, and ISRUC respectively, and the retained subsets are fixed throughout search and final evaluation. Motor imagery training and validation share subjects while test subjects are disjoint. These are reasonable controls, but they mean the reported averages describe this prepared workload.

The paper is an arXiv preprint, and the authors state that code, configurations, and supporting materials will be released. Until then, the strongest defensible claim is the one the numbers support: under a shared 40-epoch protocol across 14 datasets, an LLM-guided search produced an architecture that beat the best baseline on average test balanced accuracy by 0.29 percentage points, while losing to that baseline on weighted F1 and Cohen's Kappa.

Metric choice decides whether the search wins

The most useful takeaway is a comparison rule rather than a verdict. If balanced accuracy is the metric that matters for a deployment, AutoBCI's champion is the better of the two leading options in this study, by 0.29 points. If weighted F1 or Cohen's Kappa matters more, REVE remains ahead, and the case for the search weakens accordingly.

The second lever is pruning aggressiveness. PEEK's 38.1% error reduction and the 91.7% winner retention at three candidates per round justify using early curves to skip full training, but the weaker top-two retention warns against pruning harder than the paper tested. The paper's own priority is therefore clear: use the forecaster to save epochs, keep enough candidates alive, and judge the discovered architecture on the metric your application actually scores.

Sources used in this answer