Can scalable verifiable training environments improve long-horizon program reproduction agents?

Rebuilding a program from behavior alone

Program reproduction differs from the coding tasks most agents are trained on. In issue-resolution benchmarks the agent receives a repository and a bug report; in repository-generation benchmarks it receives a requirements document. In program reproduction, as defined by ProgramBench, the agent gets a reference program and its usage documentation but no source code and no network access. It must discover the remaining behavior by running the reference program and then design an implementation from scratch.

That setup removes the two crutches agents usually lean on: an existing codebase to patch and an explicit specification to follow. The paper notes that none of the models in the original ProgramBench evaluation fully solved a task, which is why training needs interactive environments that supply reliable behavioral supervision. Issue-resolution resources such as SWE-Gym and SWE-smith do not provide that kind of supervision, and the paper describes MindForge as restricted to command-line programs with unverifiable supervision.

Why environments and workflows both break down

The paper frames the problem as two distinct bottlenecks. The first is scaling verifiable training environments. Building a reference executable and a runtime sandbox requires accommodating repository-specific toolchains and dependencies, and the black-box tests must cover diverse functionality while rejecting incorrect reproductions. A single test-generation attempt may miss important behaviors or produce weak assertions.

The second is sustaining long-horizon development. General-purpose agent harnesses support tool use but do not by themselves keep exploration and implementation coordinated. An agent can start implementing before it has explored the reference program enough, and it can forget previously observed requirements as its context grows. The paper's response is an explicit workflow that preserves behavioral evidence and coordinates implementation, review, and revision.

These bottlenecks explain why the contribution is split into Program-Env and Program-Agent rather than a single model change. Environment quality determines what supervision exists; workflow determines whether an agent can use that supervision across a long task.

Feedback loops that repair builds and tests

Program-Env constructs environments from repositories through a feedback-driven loop. For the executable and sandbox, it separates online dependency preparation from offline compilation and returns build diagnostics to guide repairs. The paper's case study shows the loop concretely: a first attempt installs libopenimageio-dev but not the OpenEXR development package, so the offline build fails when CMake cannot locate OpenEXRConfig.cmake. The feedback packages the scripts, the error log, and a remediation instruction; the agent attributes the failure to the missing libopenexr-dev package, adds it to the apt-get install list, and the revised scripts pass online preparation, offline compilation, and the dynamic linking check.

Test construction runs across functional modules in parallel. A dispatcher decomposes the repository into modules under a complexity-based budget, instruments the executable for coverage, and extracts documentation. Each module goes to a subagent in its own workspace, with several workspaces running in parallel. Coverage instrumentation uses a rule-first, agent-fallback strategy: a language-specific rule builds a candidate, a validator checks for build artifacts and coverage metadata, and only a validation failure triggers an LLM agent to diagnose and repair the instrumentation.

Test quality is enforced by linter rules that reject weak oracles. The rules flag patterns such as a membership assertion using a generic keyword like success or error, a sole assertion checking only len(x) > n, tests that import the source module, tests that reference the source tree at runtime, and tests with network, time, or unseeded random dependencies. Feedback after each round is structured to answer three questions: which existing tests need repair, which uncovered code the next round should target, and through which public entries that code can be reached.

The output of this pipeline is 1,648 training environments spanning C, C++, Go, and Rust. Each environment provides a reference executable, a runtime sandbox, documentation, and black-box behavioral test suites for training supervision.

Separating analysis, implementation, and review

Program-Agent addresses the long-horizon problem with a role-isolated iterative workflow of analysis, implementation, and review, combined with revision management for compilation checks and failure recovery. The isolation matters because the failure mode the paper identifies is premature implementation: an agent that starts writing code before it has recorded enough behavioral evidence, then loses that evidence as context grows.

The workflow first records evidence, then implements against it, then reviews. Revision management handles compilation checks and recovery from failures, so a failed build becomes an input to the next revision rather than the end of the attempt. The paper reports that extra Program-Agent iterations continue to recover program behavior, and that Program-Agent rather than supervised fine-tuning alone reshapes the agent toward exploration and revision.

Environment gains and workflow gains are separate

The experiments use ProgramBench. Fine-tuning Qwen3.8-27B on trajectories collected from Program-Env raises its performance from 34.87% to 42.84%. That comparison isolates the value of the constructed environments: same base model, same benchmark, supervision added.

Program-Agent then raises the base and SFT models to 55.71% and 56.25% respectively. Both figures exceed some flagship models, according to the paper. The two numbers should not be merged into one headline: 42.84% measures what environment-based fine-tuning adds to the base model, while 55.71% and 56.25% measure what the workflow adds on top of the base and fine-tuned models. A reader comparing 42.84% with 56.25% is comparing a fine-tuning result against a workflow result, not two versions of the same intervention.

The training setup behind the fine-tuning number is worth noting for scale. The paper issues task-level rollout requests for all 1,648 Program-Env tasks, and because Claude Code resets or switches its conversation context during a rollout, a single task-level rollout can yield multiple interaction segments with separate contexts. Each segment is treated as a separate training trajectory, giving 64,505 training trajectories in total. The paper states explicitly that this trajectory count does not correspond to 64,505 independent environment rollouts.

What the reported gains do not establish

The headline percentages are ProgramBench results for Qwen3.8-27B and its fine-tuned and workflow-augmented variants. The paper does not report that these gains transfer to other base models, other benchmarks, or program-reproduction tasks outside the ProgramBench setup, and the claim that the augmented models exceed some flagship models is stated without a full per-model comparison in the evidence available here.

The environment pipeline also has a stated ceiling on test coverage. Test generation targets a functionality coverage threshold of 80%, runs between 5 and 20 rounds depending on repository difficulty, and stops early when core coverage gains less than 0.25% and linter pass rate gains less than 20% over a three-round window, with the earliest possible stop at round 5. The paper acknowledges that some targets are unreachable through black-box testing, which is why the early stop exists. Coverage below the threshold is therefore a property of the environment, not necessarily a failure of the agent trained in it.

The trajectory count is another place where scale can be misread. The 64,505 training trajectories come from segments of rollouts over 1,648 tasks, not from independent environment rollouts, so the effective diversity of training experience is smaller than the trajectory number alone suggests.

Credit the workflow, not only the fine-tuning

The paper's own decomposition points to a specific priority for readers evaluating this line of work. Environment construction alone moves Qwen3.8-27B from 34.87% to 42.84%, an improvement of roughly eight percentage points. Adding the role-isolated workflow moves the base model to 55.71% and the fine-tuned model to 56.25%, both above the fine-tuning-only result. The workflow contributes the larger share of the reported final performance, and the paper states that Program-Agent rather than SFT alone reshapes the agent toward exploration and revision.

That ordering matters for anyone deciding where to invest. If the goal is a better program-reproduction agent, the evidence here suggests the coordination of analysis, implementation, and review is not a cosmetic wrapper around a fine-tuned model; it is where much of the benchmark gain appears. The environment pipeline remains the prerequisite, since it supplies the behavioral supervision and the trajectories, but the final comparison favors treating the workflow as a first-class component rather than an add-on.

Sources used in this answer