How much would conference accept/reject decisions change under a different reviewer set?

The estimator targets a counterfactual, not observed disagreement

The question is not whether reviewers within one committee disagree. It is how often the final accept/reject decision would differ if the same paper were handled by an entirely different set of reviewers. The only direct measurements of that counterfactual come from two randomized experiments that duplicated review: NeurIPS 2014 (166 papers, 25.9% disagreement, accept precision 49.5%) and NeurIPS 2021 (882 papers, 23.0% disagreement, accept precision 50.6%). Because rerunning a program committee is prohibitively expensive, the paper builds an observational substitute from public review scores alone.

The estimator has three parts. A Bayesian ordered-probit model decomposes each review score into a latent paper quality and a reviewer-noise residual, with year-specific cutpoints absorbing changes in the rating scale. A logistic model maps paper-level score summaries to accept/reject, approximating the area-chair stage (5-fold cross-validated AUC 0.933 on 28,476 papers with a known decision). Then two independent committees of k reviewers are simulated per paper from the posterior (B=1,000 draws; k=2, 3, 4), and the disagreement rate and accepted-paper flip probability are recorded.

The distinction matters because a within-committee disagreement rate and a two-committee decision-flip rate are different quantities with different denominators. The paper reports the second, and it is the second that the NeurIPS experiments measured.

Two independent calibrations anchor the simulation

External calibration compares simulated quantities against the randomized experiments at matched reviewer counts. At the NeurIPS 2021 caliber of k=3, the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] against the reported 23.0%, a bias of +0.3 percentage points. Accept precision is simulated at 55.6% [51.9%, 59.3%] versus the reported 50.6% (+5pp), and committee mean-score correlation at 0.618 [0.592, 0.645] versus 0.575/0.586 (+0.04). The 2014 anchor (25.9%, n=166) sits 2.6pp above the same simulated quantity. The preregistration had expected downward bias from omitting area-chair noise; the observed bias is mildly upward but small.

Internal calibration is model-free and therefore does not depend on assumptions about how scores are generated. On 18,740 ICLR papers with four or more reviews, reviewers are randomly split into two groups of two, each group is passed through the same decision model, and the resulting split-half disagreement is compared with the k=2 simulation over 200 repetitions. The pooled split-half rate is 29.6% [29.2%, 30.0%], and the model agrees within 1pp in 2018 and in every year from 2021 to 2025. In 2017, 2019, and 2020 the model sits 3-5pp below the anchor; the paper attributes this to stronger reviewer-assignment homogeneity in the early years and to the coarser 2020 scale, but presents it as a hypothesis rather than a tested explanation.

Both anchors are numerical comparisons rather than refits. The public NeurIPS 2021 reviews cover accepted papers only, so the external check matches reviewer-count calibers instead of rerunning the full pipeline on NeurIPS data.

Paper quality explains roughly two-fifths to three-fifths of score variance

The signal-to-noise ratio rho is the share of latent score variance attributable to paper quality, with the residual variance fixed at 1. Across the nine ICLR years it ranges from 0.376 [0.356, 0.396] in 2021 to 0.583 [0.535, 0.629] in 2017, with the other years between 0.410 and 0.481. That magnitude is consistent with the NeurIPS 2014 variance-decomposition estimate of roughly 50% subjective variation.

This is the quantity most easily confused with the disagreement rate. Rho describes how much of the variation in scores reflects the paper rather than the reviewer; the disagreement rate describes how often two simulated committees reach different decisions. A year can have moderate rho and still produce many flips if acceptances cluster near the decision threshold, which is exactly what the paper finds for 2021.

Reviewer identities are unavailable in the public data, so reviewer stringency cannot be estimated separately and is folded into the noise term. The model cannot distinguish a harsh reviewer from a noisy one, a limitation declared in the research plan.

Smaller committees flip more decisions

At k=2, simulated two-committee disagreement ranges from 22.8% in 2017 to 30.3% in 2025. At k=3 it ranges from 19.7% to 26.7%, and at k=4 from 17.7% to 24.2%. Moving from two to four reviewers reduces disagreement by about 5 percentage points, consistently across years.

Among accepted papers, 30-50% would be rejected by an independent committee at k=2. That flip probability is highest in 2020 (47.9%) and 2021 (50.2%), well above the 30-41% range of the other years. Across years the flip probability correlates -0.74 with rho and -0.83 with the acceptance rate, though the two predictors are collinear and an OLS fit on nine observations is not significant (R2=0.70).

The practical reading is that committee size buys stability, but not enough to make decisions near-deterministic. Even at four reviewers, roughly one in five papers would receive a different decision from a second committee.

2020 and 2021 produced the same symptom for different reasons

2020 used a four-point scale {1,3,6,8}, and the compression fingerprints are visible: 23.7% of papers had zero within-paper score variance, against 4.7-13.1% in other years; score entropy was the lowest in the sample at 1.27; and cutpoint spacing relative to the paper signal was the widest at 1.67 sigma. With k=2, a single reviewer moving one notch shifts the committee mean by 0.75 or more, making threshold crossings more likely.

2021 shows no compression fingerprint; its zero-variance share is the nine-year minimum at 4.7%. Its mechanism is low signal plus threshold crowding: rho is the sample minimum at 0.376, and 30.4% of accepted papers sit within 0.25 of the decision threshold, the sample maximum, with a median margin of 0.50, the sample minimum. High noise combined with many acceptances hugging the threshold makes flips easy for a second committee.

A counterfactual isolates the scale effect. Mapping 2021's ten-point scores to the 2020 caliber by nearest neighbor, refitting the ordered probit, and re-simulating with the acceptance rate fixed at 33.1% in both worlds lowers rho from 0.376 to 0.345 and raises disagreement by about 7pp (k=2: 0.301 to 0.374) and the accepted-paper flip probability by about 11pp (k=2: 0.455 to 0.565), consistently across k=2, 3, and 4. This cross-validates against the observed data: 2020's actual rho (0.439) exceeds 2021's (0.376), meaning 2020 had less latent noise, yet the coarse scale inflated its observed disagreement to 2021's level (0.268 vs 0.260) and pushed its flip rate higher (0.479).

The counterfactual runs in one direction only, from 2021 to the 2020 caliber, and uses a threshold decision rule with randomized tie-breaking rather than the main logistic model, because a fixed 0.5 logistic cutoff collapses the counterfactual acceptance rate to 9.9%. The effect size is therefore specific to that threshold-rule caliber.

Nine years of growth show no attributable trend in reviewer noise

The preregistered longitudinal regression (OLS, n=9) finds no predictor significant at p<0.05 for any of the three outcomes. For k=2 disagreement: log submissions +0.0338 (p=0.056), reviews per paper -0.0581 (p=0.103), four-point-scale indicator -0.0327 (p=0.207), six-point-scale indicator +0.0140 (p=0.489). Submission growth, the 2021 shift from three to four reviewers per paper, and scale changes moved together, with a VIF of about 5.8 for log submissions against reviews per paper, so their contributions cannot be separated.

A formal trend test of rho on year gives a slope of -0.0122 per year with p=0.110. The power calculation clarifies what this null means: with nine observations, the design has 80% power only for slopes of at least ±0.022 per year, about ±0.20 over the nine-year window, and power at the observed slope is 35%. Bivariate disagreement-on-year trends are nominally significant (k=2: +0.0062/year, p=0.028; k=3: +0.0065/year, p=0.008) but fragile: dropping 2017 raises the k=2 p-value to 0.147, and dropping any of 2023-2025 raises it to 0.052-0.080.

The paper's conclusion is that there is no robust, attributable time trend in reviewer noise. The nominal rise in k=2 disagreement from 22.8% in 2017 to 30.3% in 2025 cannot be separated into contributions from submission volume, reviewer counts, or scale changes. This is an honest null, not proof of stability.

The 2023 breakpoint test cannot support an LLM attribution

An interrupted time series on normalized within-paper score variance over all nine years finds no break at 2023: level change +0.0099 (p=0.864), slope change -0.0026 (p=0.766). The design has almost no power for this test; at 80% power it can only detect level jumps of ±0.19 against a baseline variance of 0.01-0.04, meaning five-to-tenfold changes. Dropping the anomalous 2020 still finds no break (level p=0.191, slope p=0.201), though the pre-2023 baseline slope is itself positive (+0.0012/year, p=0.031), so within-paper variance was slowly rising before 2023 without a jump at the breakpoint; 2023-2025 values are stable at 0.017-0.019.

Two data constraints, both verified before analysis, reshaped this question. The primary data carry no review text (0 of 134,912 reviews non-empty) and no confidence values, and the only review-text source, PeerSum with 53,732 reviews, ends in 2022. The preregistered aggregate LLM-share estimate therefore could not be run, and no per-review AI classification was performed.

What can be reported is descriptive and limited to 2018-2022. Review length rose about 32% from 2018 to 2021 (mean 2,316 to 3,047 characters) and then flattened in 2022 (3,065); mean confidence is stable at 3.4-3.8 with a median of 4.0 every year. None of these proxies can be attributed to LLMs; they may reflect submission growth, process changes, or scale changes. The paper's own summary is that the data cannot support a conclusion about LLM effects on reviewing in either direction.

The estimator is validated on ICLR and applied only to ICLR

The variance decomposition uses 35,617 papers with at least one scored review; 496 papers without scores are excluded. Withdrawn papers with scores enter the variance decomposition but are excluded from decision-model fitting, which requires an observed outcome. Of the 36,113 papers, 11,213 were accepted, 17,426 rejected, and 7,474 withdrawn with unknown decision, all verified as withdrawals rather than data corruption.

The internal anchor's sampling frame is close to the full population: anchor papers average 5.47 versus 5.40 mean score, 40.1% versus 39.2% acceptance, and 4.17 versus 3.75 reviews per paper. Cross-venue generalization was moved out of the main paper, so the estimator is validated against ICLR and NeurIPS anchors but applied only to ICLR.

The most decision-relevant comparison for a reader weighing review reform is the scale effect rather than the time trend. Coarsening the rating scale from ten points to the 2020 four-point caliber raised simulated disagreement by about 7pp and the accepted-paper flip probability by about 11pp with cohort, year, and acceptance rate held fixed, while destroying about 8% of the signal share. That is a larger and better-identified lever than anything the nine-year trend analysis could attribute.

Sources used in this answer