How does SWE-Duel create a self-scaling arena for coding agents?

Static benchmarks decay from the day they ship

The paper starts from a structural complaint about coding benchmarks. A benchmark released as a fixed task set is built from openly accessible sources, so its content is eventually absorbed into training corpora, and stronger agents eventually saturate it. The authors cite OpenAI's decision to stop reporting results on SWE-bench Verified after finding that frontier models could reproduce its gold patches as an illustration of this failure mode.

Their proposed alternative is a benchmark that is self-scaling: tasks and oracles are created at evaluation time, so no content exists in advance to leak, and difficulty is set by the participating agents rather than fixed at release. The paper also distinguishes this from continuously updated benchmarks such as SWE-bench-Live and SWE-rebench, which reduce leakage by collecting tasks created after training cutoffs but still bound difficulty by the daily issues human developers face. Model-generated task approaches such as BugPilot and SwingArena move closer, but BugPilot generates a fixed task set ahead of time and SwingArena still draws tasks from a fixed pool of human-written GitHub issues, generating only the oracles at evaluation time.

Red authors a trap, Blue must repair it

Each turn is a mirrored duel on a real repository. The Red agent works across three context-isolated sessions: it proposes a new feature, implements it together with feature tests, then embeds a subtle bug in that feature along with sealed bug-triggering tests. The resulting pull request is admitted into the duel only under two conditions. First, it must pass the feature tests and the repository's existing regression tests while failing exactly the bug tests. Second, a fresh session of the same Red agent must be able to repair the bug blind, which rules out tasks that are unsolvable in principle.

The Blue agent then reviews the pull request without being told that a bug exists. It wins the round only if its revision passes all three test suites: feature tests, existing repository tests, and the sealed bug tests. After that, the two agents swap roles. Because every feature, bug, and oracle is created at evaluation time, those specific tasks cannot appear in earlier training data, and because the tasks are authored by the agents under evaluation, difficulty is not capped by a human-written pool.

The solvability gate caps how hard tasks can get

The blind self-repair requirement is what keeps the arena honest, but it also ties task difficulty to the author's own solving ability. A Red agent cannot admit a bug it cannot fix itself in a fresh session, so the hardest admissible defect is bounded by that agent's capability. The authors acknowledge this directly and sketch a future extension: a solvability check that uses privileged information, such as telling the self-review session that a bug exists or which type to look for, would still prove the bug is fair and solvable while giving the author leeway to embed subtler bugs that no longer need to be caught by a blind self-review.

This matters for reading the tournament results. A high Red win rate measures how often an author's admitted tasks defeat opponents, not how hard the author could make tasks in an unconstrained setting. The paper's own framing treats the gate as a design trade-off rather than a solved problem.

Eleven participants, 1,500 turns, two win styles

The main experiment is a round-robin tournament among 11 model-harness participants on 15 repositories in five languages, for 1,500 turns. Each pair plays one match, and within a match each participant plays Red and Blue equally often on the same repositories. A participant's match score sums its points in both roles; Blue earns a point when its revision passes all three suites, Red otherwise. If a participant admitted fewer than the scheduled number of tasks on a repository because its slots were exhausted, each missing task is scored as a Blue win without running Blue, which lowers that author's match score.

The leaders win in different ways. GPT-5.5 with Codex CLI ranks first as the strongest solver, repairing 98.3% of the defects it reviewed. Claude Opus 4.8 with mini-swe-agent ranks second as the strongest task author, with a 27.3% Red win rate. The paper's turn-level analysis, which keeps every turn as an independent Bradley–Terry outcome, puts Claude Opus 4.8 with mini-swe-agent first at +0.30 and GPT-5.5 with Codex fourth at +0.18, reflecting the per-turn weight on Opus's field-best Red win rate. The authors note that rank uncertainty is larger in the turn-level analysis, and that the top seven intervals overlap.

Missing task slots can outweigh win rate

The full-schedule scoring treats an empty Red slot as an automatic Blue win, so a participant that admits few tasks concedes points without playing. The contested-only refit drops those auto-win turns and re-weights records to contested-only Red wins, and it moves the rankings substantially: GPT-5.5 with mini-swe-agent goes from third to first, GPT-5.5 with Codex from first to second, Claude Opus 4.8 with mini-swe-agent from second to fifth, Nemotron-3-Ultra from ninth to third, and MiniMax-M3 from tenth to seventh.

The paper attributes these moves mainly to the task-generation coverage penalty. Nemotron-3-Ultra admitted 7 of 15 slots, so each of its ten matches conceded eight uncontested Blue points, totalling 80 auto-win turns across the main tournament; deleting exactly those concessions produces its +0.145 refit coefficient and its jump to third. That position is fragile: its clustered interval [−0.35, +0.75] touches zero at the lower bound, and nine of its ten contested Red wins arise from a single admitted task. The authors also report that the full-schedule turn-level ranking is much closer to the main tournament's full-schedule match-level ranking (Kendall's τ = 0.782) than to the contested-only ordering (τ = 0.382), so the auto-win policy, not the aggregation granularity, drives the sensitivity.

Newer model versions write harder tasks

The paper's most direct evidence for self-scaling comes from a within-family comparison. Tasks written by the newest version of a model family defeat a fixed-reference solver far more often than tasks written by older versions: 42.9% versus 4.0%. The authors present this as preliminary evidence that task difficulty scales with task authoring capability, which is the property a self-scaling benchmark needs.

The evidence is explicitly preliminary, and the paper's own analysis shows why a single number should not be over-read. In the concentrated Red-win case, nine of Nemotron-3-Ultra's ten contested Red wins come from one task on the Gogorilla/csrf repository, where the sealed defect is a one-condition change in a prefix matcher and every visible signal corroborates the defective behavior. A manual audit found the sampled solving run was a genuine failure rather than an unsound oracle, but most solvers did locate the exact condition, so the task does not cleanly test whether Blue can find a hidden bug. Five of that author's seven admitted tasks defeat no one, which the authors describe as one exceptionally well-constructed defect plus six ordinary tasks.

New entrants join with logarithmic new matches

The arena is designed to grow. To add a new participant to a finished tournament of M participants, the entrant first creates its own task bank and then plays against only ⌈log2 M⌉ + 1 incumbents, selected with the active sampling rule of Chatbot Arena. Incumbents' task banks and all earlier match results are reused unchanged, so the number of new matches grows logarithmically with M. In the reported late entrance, Gemini-3.1-Pro with mini-swe-agent joined with five matches without changing the existing order, and its marginal entrance cost was $243.40 against the main tournament's $2,109.21 total API spend.

The paper is careful about what its cost and efficiency numbers mean. The associations between Bradley–Terry strength and API cost or output-token use are described as observational, combining model pricing, generation coverage, and behavior, not a controlled efficiency comparison. A separate output-token-budget-matched ranking at 638,976 tokens per participant, which excludes Red task creation, puts GPT-5.5 with Codex first at +2.60 and GPT-5.5 with mini-swe-agent second at +1.38, and the authors state it is not meant to replace the main tournament ranking.

Solver strength and author strength are separate axes

The tournament's most useful lesson is that a single leaderboard hides two different skills. GPT-5.5 with Codex CLI leads as a solver, repairing 98.3% of the defects it reviewed, while Claude Opus 4.8 with mini-swe-agent leads as an author with a 27.3% Red win rate; the full-schedule match ranking and the turn-level ranking order these two differently. Which one ranks first depends on whether the estimand weights per-match outcomes or per-turn strength, and the paper reports both rather than collapsing them.

For readers deciding what to take from SWE-Duel, the relevant split is between using it as a solver benchmark and using it as evidence about task-generation capability. The within-family 42.9% versus 4.0% result supports the second use, but only as preliminary evidence, and the concentrated Red-win case shows how much a single well-constructed task can move an author's record. The paper's own priority is therefore the self-scaling mechanism itself: tasks and oracles created at evaluation time by the agents under test, with the blind self-repair gate as the condition that keeps those tasks fair and solvable.

Sources used in this answer