Can AI Systems Match Human Experts in Experimental Research Taste?

TasteVal scores experiment design, not problem choice

The paper defines research taste as the ability to pick interesting problems, design experiments, and interpret experimental results. TasteVal deliberately measures only the middle and last parts: given a fixed research problem, it asks how well a model iteratively designs experiments and draws conclusions from their outcomes. Problem selection is explicitly outside the benchmark.

That boundary matters for reading every number below. A high TasteVal score says a model can steer an experimental campaign toward a known scoring signal. It does not say the model would have chosen that problem, or that the problem resembles frontier AI R&D. The authors state that progress so far has come from shallow and moderate optimization rather than novel ideas.

Compute efficiency is the taste metric

TasteVal operationalizes experimental taste as compute efficiency. A Researcher who reaches the same score as an expert human using half the serial experimental compute has twice the experimental taste. Taste is therefore expressed as a multiplier on experimental compute, which the authors treat as a key input to forecasts of AI progress.

This is not the same as final performance. The paper reports two distinct quantities: a compute multiplier, which is how much less serial experimental compute a model needs to match a reference score, and a performance multiplier, which is measured by normalized final performance. The two behave differently over time, so a reader should not substitute one for the other.

A fixed Coder isolates taste from coding

To separate taste from implementation ability, the model under evaluation acts as a Researcher that iteratively designs experiments, while a fixed Coder agent implements them and reports their results. The Researcher never reads the Coder's workspace or transcript; it sees only a report containing an index.md plus assets, and it can watch training and validation curves while an experiment runs. Test scores are withheld from the Researcher at every layer.

The Coder is bound by rules that protect the measurement: it must not modify experiments, must not fill in significant gaps, must not run unsolicited variants or ablations, and must not draw conclusions in its report. An Experiment Monitor reviews each description before execution and rejects low-level implementation detail, dictated file structure, underspecified plans, or anything that would reveal test data. A rejected experiment consumes no GPU hours.

The Researcher executes until either the 40 H100 hour or 120 wall-clock hour budget is exhausted. A Liaison channel lets the Researcher ask about the Coder's status; models used it less than humans, with 69% of model runs asking at least one question (median 4 among those runs) versus all 24 human runs (median 28 per run).

Eight tasks, 24 experts, and a best-attempt baseline

TasteVal consists of 8 novel, challenging, open-ended tasks representative of frontier AI R&D, spanning curating pre-training data, pre-training language models, fine-tuning pre-trained models, modeling human preferences, and aligning models against adversarial prompts. The tasks are not released, to keep the benchmark uncontaminated.

Each task has two baselines. A naive baseline is a weak submission with zero taste that non-trivial submissions are expected to beat. The expert baseline comes from 24 recruited baseliners with research experience directly related to their tasks, who had recently worked at organizations including OpenAI, Google DeepMind, NVIDIA, Microsoft Research, and the University of Oxford. Each was paid to research the relevant field for 4–16 hours and write a short literature review before starting.

At least two human experts attempted each task under the same compute and wall-clock budgets and with the same Coder (Opus 4.8). The per-task expert baseline is the best human attempt per task, not the average. Baseliners were not allowed to use AI assistance for designing experiments or interpreting results.

Opus 5.5 beats the expert baseline at 2.3x

The study evaluates 20 models released between 2023 and 2026. The best-performing model, Opus 5.5, exceeds the expert baseline with a compute multiplier of 2.3x (95% CI 1.15–4.37), at roughly 1/30 of the baseliners' average per-run cost. The confidence interval is wide and its lower bound sits close to 1x, so the margin over the expert baseline is not tightly pinned down.

The authors also note that their expert baseline had a substantially larger number of experiments rejected per run on average than recent frontier models. Classifying the monitor's rejections of human-proposed experiments using Fable 5.1 (max), 75% were due to underspecified descriptions, 10% from accidental submissions of incomplete descriptions, 9% from low-level implementation details, and 6% from a long tail of errors. Across all human runs, rejections per proposed experiment were 69% lower in the last 10 GPU-hours than in the first 10. Because rejected experiments consume no GPU hours, the authors do not interpret the large number of rejections as clear evidence that the expert baseline was under-elicited.

The trend break separates taste from performance

On TasteVal, the compute multiplier of frontier models has doubled approximately every 3.0 months since December 2025 (95% CI 1.7–5.0), up from every 14 months between 2023 and December 2025. The break is placed at the release of GPT-5.2, and the two separate log-linear fits reach R2 = 0.96. Error bars come from a two-level hierarchical bootstrap with 1,000 resamples, drawing first over tasks and then over runs within each task.

Measured by normalized final performance, frontier models show no trend break, doubling every 14.6 months. That contrast is the paper's sharpest internal comparison: the acceleration appears in compute efficiency, not in the final scores models reach. The authors also report that truncating runs before the full 40-hour budget leaves the model ranking and frontier multipliers nearly unchanged, with a rank correlation of 0.94 with the 40-hour ordering at 5 hours and at least 0.97 from 10 hours. The post-break doubling time shortens as the budget shrinks, from 14.6 months at 40 hours to 11.6 months at 10 hours and 10.5 months at 5 hours, and at every budget lies within the wide 40-hour 95% interval of 8.4–29.8 months.

Where these results do not transfer

The authors group limitations by whether they mainly affect the level of the compute multiplier or its rate of growth. Several features may overstate the level. TasteVal tasks are deliberately easy to verify, with scoring that has low noise and typically takes under 10 minutes on both splits, and they require only a single H100 for development, far below frontier ML development scale. The tasks also lack messiness factors central to frontier ML development: every task has a clean ground-truth signal of progress, and none requires real-time coordination with other agents. On METR's 16-factor messiness rubric the authors self-assign a mean of 7.5, compared with about 3.7 for METR's Time Horizon v1.0 tasks, but they note these scores are self-labeled and that the missing factors are among the most important for frontier research.

Other features may understate the level. Elicitation of both the Researcher and Coder was limited, and optimizing scaffolds or updating the model driving the Coder could substantially increase performance. Cost also cuts the other way: the best model, Opus 5.5 at max reasoning, costs an average of $282 per run in API cost, roughly 1/30 of the expert baseline's per-run cost.

The paper also states plainly that a large gap remains between TasteVal and frontier AI R&D, and that TasteVal does not measure problem selection.

What the multiplier means for AI R&D forecasts

The paper's own framing is that taste parameters in AI R&D forecasts have until now been set by expert judgment, and TasteVal provides a direct measurement that can be repeated as new models are released. In the paper's illustrative calculation, a doubling every 4.6 ECI units raises effective experimental compute by ln 2 / 4.6 = 0.151 per ECI point, scaled by the share of research spending on experiments that AI directs. Under those assumptions, algorithmic progress is self-sustaining whenever the return to researcher quality, gamma, exceeds about 0.21, for almost any split of spending between experiments and AI researchers. Cunningham et al. estimate gamma at 0.15–0.3, so this holds for most of their range. The result depends on full automation: if a third of research spending still goes to human researchers, the loop is self-sustaining only in the upper half of that gamma range, and not at all beyond that.

For a reader deciding what to take from this paper, the operative distinction is between the compute multiplier and the performance multiplier. The 3.0-month doubling since December 2025 is a statement about how much serial experimental compute a frontier model needs to match a reference score on 8 easy-to-verify, single-H100 tasks, with a fixed Coder and a best-attempt human baseline. The 14.6-month doubling in normalized final performance is the flatter, break-free trend. The paper's most decision-relevant claim is that these two curves have diverged, and that the divergence is measured on tasks the authors themselves describe as easier to verify and less messy than frontier AI R&D.

Sources used in this answer