[ICLR 2025] CodeScout: Solving the "Vague Bug Report" Problem for AI Agents

TL;DR

Even the most powerful AI agents struggle when given a one-liner bug report. CodeScout introduces a systematic pre-exploration phase that "looks before it leaps," transforming vague developer queries into repository-aware, actionable guides. By extracting structured insights from a codebase before the agent starts its loop, CodeScout boosts resolution rates by 20% on SWEBench-Verified.

The Problem: The Specification Gap

Why do state-of-the-art agents like SWE-agent or OpenHands fail? Recent analysis suggests it isn't always a lack of reasoning; it's a lack of context.

1. Information Asymmetry: Natural language bug reports often omit reproduction steps or technical context that humans assume is "obvious."
2. Reactive Exploration: Without a global view, agents fall into loops of "over-exploration" or "stubborn fixing"—trying the same wrong solution repeatedly because they don't understand the underlying architecture.

Methodology: The Architecture of Pre-Exploration

CodeScout operates as a plug-and-play pre-processor. It doesn't modify the agent's logic but rather improves its "intelligence briefing."

1. Repository Knowledge Graph (G)

Instead of treating the codebase as a flat list of files, CodeScout uses an AST (Abstract Syntax Tree) visitor to build a directed graph of class inheritances, function signatures, and import dependencies.

2. High-Level Scoping & Fine-Grained Analysis

The system identifies up to 15 key "exploration targets" (files/classes). For each, it generates:

• Role Assessment: How this code likely causes the bug.
• Fix Hints: Predicted locations for modifications.
• Alternative Hypotheses: Thinking outside the box for root causes.

Figure 2: The CodeScout pipeline—moving from raw codebase to augmented problem statement.

3. Problem Synthesis

The final output ($P_{aug}$) is a "supercharged" bug report containing technical insights, reproduction steps with internal details, and specific exploration hints.

Experiments and Results

The authors tested the framework on SWEBench-Verified using a variety of LLMs including DeepSeek R1, Qwen3 Coder, and GPT-5-mini.

• Resolution Gains: Resolution rates increased consistently. For Qwen3 Coder, the number of resolved issues jumped from 183 to 207 (+13.1%).
• The "Stronger Augmenter" Effect: This is the most fascinating finding. Using a strong model (Qwen3) to prepare the problem statement allowed a weaker model (DeepSeek R1) to resolve 164 issues, up from its baseline of 108—a massive 51.9% boost.
• Targeted Exploration: Trajectory analysis showed that agents spent less time the "find" and "grep" tools and more time on "patching" and "testing" when using CodeScout, proving that the pre-exploration replaced wasted agentic cycles.

Table 1: Performance gains for various LLMs using CodeScout components.

Critical Insights: Why It Works

The "Agentic Intra-trajectory Augmentation" ablation is key. When agents were simply told to "think better" or "augment yourself" during the task, performance actually dropped. This suggests that separating understanding from execution reduces the cognitive load on the LLM.

By spending a few cents on pre-computation, CodeScout saves dollars in wasted agent trajectories. It provides a "strategic mental model" that current reactive agents lack.

Conclusion & Limitations

CodeScout proves that the bottleneck for software agents is moving from "model capacity" to "input quality." However, the study is currently limited to Python and single-repository tasks. The future of this field lies in scaling these "knowledge graphs" across massive, multi-repo enterprise architectures where context is even scarcer.

Takeaway for Devs: If you want your AI agent to work, stop giving it one-line prompts. Give it a map.