Can built-in FAIL TO PASS tests measure runtime performance of correct patches?

Correctness tests leave runtime cost unmeasured

Repository-level issue-resolution benchmarks such as SWE-bench judge a submitted patch mainly by whether it passes the correctness tests, especially the FAIL TO PASS (F2P) tests. An F2P test fails on the unfixed repository and passes only after the repair, so it encodes the behavior the fix is supposed to change. The benchmark does not require the developer's exact implementation, which means several genuinely different patches can pass the same F2P tests while executing at very different costs.

That gap is the paper's starting point. Passing tests says nothing about how efficiently a patch runs, and the benchmark provides no performance workload of its own. Designing one per instance would be costly, so the authors ask whether the F2P test already present in each instance can be repurposed as a free performance workload.

The F2P test works as an approximate workload

The argument for reuse is that the F2P test exercises exactly the code path the fix touches. Running it repeatedly under instrumentation yields execution metrics for each patch without any extra artifact. The authors note a caveat: an F2P test usually bundles data preparation, the invocation of the tested behavior, and assertions in one body, and only the invocation is patch-sensitive. If preparation and assertion cost the same for every solution, they shift all solutions by the same amount, which dilutes measured differences without favoring any patch.

Run-to-run noise is handled by repeated execution rather than by a single reading. The paper states that the ordering among solutions remains trustworthy when a difference is statistically significant, and that noise only clouds solutions whose costs are close. The PASS TO PASS regression suite is deliberately excluded: those tests far outnumber the F2P ones, mostly do not exercise the fixed code path, and would multiply cost with little analytical gain.

Filtering shrinks the benchmarks to measurable cases

Not every instance supports this measurement. The authors impose three constraints: the F2P field must contain a single test so each instance yields one comparable reading; the test must sustain repeated execution within one process to suppress noise; and it must be free of input and output operations whose variance would drown out CPU-level signals. One static rule handles the first constraint and dynamic probing handles the other two, leaving 339 instances.

Four frontier models — Claude Opus 5, GPT 5.6, DeepSeek V4 Pro, and Qwen 3.8 Max — were paired with SWE-agent to solve this filtered corpus from the verified splits of SWE-bench and SWE-bench-Live. After merging, 262 cases contained at least two genuinely different correct solutions, including the developer ground-truth patch. Each solution was measured in instruction count and wall-clock time on the native F2P test.

Welch ANOVA separates real gaps from noise

Measurement runs in two loops inside a per-instance Docker environment. The inner loop repeats the F2P test function N times in the same process to lift the signal above the noise floor, with N calibrated per case so the fastest solution takes between 4 and 8 seconds; calibrated N averaged 6.8×10^5 invocations per case, with a median of 1.6×10^4. The outer loop repeats the whole process G = 5 times as independent replicates, each in a fresh process, and those five measurements form the case-level samples.

For statistics, the authors replace the relative standard deviation used in prior work with Welch ANOVA, which accounts for both between-solution and within-solution variation. This is the paper's stated methodological contribution: a criterion that flags a genuine runtime difference among correct patches rather than treating ordinary variance as a signal. The result is that at most 42% of cases show statistically distinguishable differences among their different correct patches.

Resolution rate and runtime rank models differently

The headline comparison concerns the four models. On the commonly solved subset of 211 cases, speedup against the ground truth is aggregated as a geometric mean of per-case ratios, where a value below 1 means the model patch is faster. GPT and DeepSeek lead, Claude trails them but stays faster than the ground truth, and Qwen is slower than the ground truth overall. The runtime ordering is roughly GPT ≈ DeepSeek > Claude ≫ Qwen.

That ordering sits far from the resolution-rate ordering, which the paper reports as Claude ≫ Qwen > GPT ≈ DeepSeek. The two dimensions nearly reverse. The authors read this as evidence that resolution rate alone does not characterize a model's capability at repository-level issue resolution. The ordering also stays consistent between the full set and the distinguishable subsets for both metrics, and gaps widen on the distinguishable subsets, which the authors take as support for using those subsets as a stand-in for the full corpus.

Developer patches hold no systematic speed advantage

Comparing each model against the ground truth on its own resolved cases, GPT, DeepSeek, and Claude all run faster than the developer patch on aggregate, while Qwen is consistently slower. The win and loss counts complicate the aggregate: Claude loses more cases than it wins yet stays faster overall, meaning its winning cases speed up by a larger margin than its losing cases slow down. Qwen's wins and losses are roughly balanced, suggesting its losing cases slow down more than its winning cases speed up.

The paper's interpretation is that the ground truth cannot be assumed to be the most efficient solution; it is one solution of equal standing among others. That is why the authors prefer the geometric mean over the harmonic mean used in related work that aggregates speedup against a fixed expert reference. The geometric mean keeps the comparison stable when numerator and denominator swap, so pairwise orientations remain interchangeable.

Performance analysis exposes architectural regressions

The case study adds a qualitative result that correctness evaluation cannot produce. Three repair strategies for the same issue vary in performance by more than a factor of two, and the two costly ones silently reintroduce a heavy dependency the developer had deliberately removed. The correctness tests pass for all of them; only the runtime measurement reveals that the fix undoes an architectural decision.

This is the paper's strongest argument for measuring patches rather than only scoring them. A patch that passes every F2P test can still carry a cost the benchmark never sees, and in this case the cost traces to a design choice rather than incidental overhead.

Where the measurement stops being informative

The method has stated limits. The filtering is restrictive, though the authors describe it as deterministic and reproducible. Both benchmarks draw from Python repositories, and the authors plan to extend to performance-sensitive languages such as C and C++. All solutions come from a single SWE-agent framework, so patch diversity stems mainly from the models rather than the agent; human-written ground-truth patches broaden the comparison, and the authors plan to add frameworks such as OpenHands.

There is also a measurement-level limit: when a test's early repetitions perform the repair-relevant work and later ones merely read the state left behind, measurements degenerate into indistinguishable tiny values and discrimination among solutions drops. The authors acknowledge this. The practical consequence is that a non-significant Welch ANOVA result should be read as the test failing to separate patches, not as proof that the patches perform identically.

Report runtime alongside resolution rate

The paper's priority is specific: evaluations of repository-level issue resolution should report patch runtime performance next to resolution rate, because the two orderings nearly reverse and neither substitutes for the other. The concrete threshold for acting on a single case is the Welch ANOVA verdict — only the distinguishable cases, at most 42% of the 262, support a claim that one correct patch is faster than another.

For readers comparing models, the actionable split is between the full corpus and the distinguishable subsets: the model ordering holds in both, but the gaps widen once non-discriminating cases are dropped, so a comparison that ignores the distinguishable subset understates real differences. For readers judging a single patch, the case study sets the bar higher than test passage: a patch that reintroduces a removed heavy dependency passes correctness while losing more than half its speed relative to a leaner repair.

Sources used in this answer