[ICLR 2025] Critic Rubrics: Teaching Machines to Judge Coding Agents via Real-World Failure Modes

TL;DR

Current coding agents are great at passing unit tests (benchmarks) but struggle in the messy reality of human collaboration. This paper introduces a rubric-supervised critic that learns from 154k real-world interaction segments. By predicting 24 specific behavioral flaws (like "skipping tests") alongside sparse success signals (PR merges), the authors built a 4B model that makes agents smarter, faster (83% less compute), and more reliable across different LLM backbones.

The Core Conflict: Benchmarks vs. Reality

In academia, we love verifiable rewards. If the unit test passes, the agent wins. But in the real world:

1. Signals are Sparse: Users rarely leave "5-star" reviews for every code edit.
2. Signals are Noisy: A PR might be rejected not because the code was bad, but because the reviewer was busy or the requirements changed.
3. The Credit Assignment Problem: In a conversation with 20 turns, which specific action caused the final failure?

Previous evaluators trained on "pass/fail" benchmark data fail miserably in the real world because they haven't learned the process of good engineering—only the outcome of passing a specific test suite.

Methodology: The "Critic Rubrics" Framework

The authors' breakthrough lies in changing what the critic learns. Instead of just predicting "did this PR get merged?", the critic learns to identify behavioral features.

1. Modeling Interactions as Segments

The researchers broke down long, multi-turn conversations into segments (User Request $\to$ Agent Actions $\to$ Finish). This allows for much finer credit assignment than looking at a whole GitHub Issue at once.

2. The 24-Feature Rubric

They defined a taxonomy of 24 behaviors across three categories:

• Agent Issues: Misunderstanding intent, looping behavior, insufficient debugging.
• User Patterns: Frustration, requests for reversion, clarification.
• Infrastructure: Environment timeouts or agent-caused disk errors.

Figure 1: The pipeline from production traces to a multi-task critic.

3. Code Survival: A Better Label

Instead of the binary "PR Merge" label, they used Code Survival. This measures the percentage of the agent's code that actually made it into the final codebase after human review. This is a much "cleaner" signal of value.

Experimental Insights: Why Rubrics Matter

The Failure of Cross-Eval

A shocking finding: Critics trained only on benchmarks are near-random on real-world data (AUC 0.45-0.48). This proves that academic benchmarks are currently a "walled garden" that doesn't capture the nuances of human-agent interaction.

Robustness Across LLMs

When training a critic only on "Success" signals, it tends to overfit to the quirks of a specific model (e.g., Claude Sonnet). However, when trained with Rubrics, the critic learns universal engineering principles.

• Success-only: Fails when switched from Sonnet to Opus.
• Success + Rubrics: Maintains performance gains across both models.

Figure 3: Correlation between rubrics and success. Note how "Incomplete Implementation" is a universal predictor of failure.

Scaling Inference: Best-of-N and Early Stopping

The most practical application of this critic is Inference-Time Scaling.

1. Best-of-N: Generate 8 attempts, let the Critic pick the best. This boosted SWE-bench scores by 15.9 points.
2. Early Stopping: If the Critic sees the agent is "looping" or "failing to debug," it kills the trajectory early. This achieved an 83% reduction in compute while still improving accuracy.

Critical Analysis & Future Work

While this is a major step forward, the "ground truth" for the rubrics still comes from Frontier LLMs (o3/Claude) acting as annotators. This creates a potential "circularity" where the critic is limited by the reasoning capabilities of its teacher.

Main Takeaway: For the industry, this paper provides a blueprint for using "unlabeled" production logs. By adding a small layer of rubric-based LLM annotation, companies can turn millions of useless traces into a high-quality training signal for their agentic workflows.

Evaluation Table Snapshot:

The rubric-supervised model is significantly more "calibrated," knowing exactly when to stop and when to try again.