Endpoint scores hide whether an agent actually learned
Most agent evaluations ask what a system can accomplish at one fixed moment. That question cannot separate an agent that started strong from one that improved, and it cannot show whether gains survive beyond the interactions that produced them. The paper therefore reframes evaluation around three questions: does future performance improve and generalize beyond the learning interactions, how efficiently are new capabilities acquired, and where does the improvement process break down?
To make those questions testable, the study freezes model weights and lets improvement happen only through persistent artifacts the agent builds itself: tools, skills, strategies, and textual memory. Each acting episode starts in a fresh context, while later instances inherit artifacts refined from earlier training interactions. This isolates what the agent has externalized from what its underlying model can already do.
Plasticity measures gain per unit of learning cost
The paper's central metric, agent plasticity, is the efficiency with which an agent converts experience into gains in future held-out performance. It is not a synonym for self-improvement. It captures one property of the improvement process: how much held-out capability an agent buys for the learning it consumes.
The measurement separates three things that are easy to conflate. Held-out in-distribution performance tests whether improvements generalize beyond the interactions used for learning. Held-out out-of-distribution performance tests transfer to harder regimes. Learning cost is tracked alongside both, so a gain is never reported without the effort that produced it. Because revisions can improve, preserve, or degrade performance, the study records complete trajectories rather than endpoints alone.
Four environments test the same protocol
The protocol runs across Chess, Go, Hex, and NetHack. Chess, Go, and Hex are fully observed, alternating-turn board games with well-defined feedback; NetHack adds a long-horizon, partially observed, single-agent setting. Only training trajectories reach the reflection loop. Held-out in-distribution and out-of-distribution splits remain evaluation-only, so they cannot feed back into the artifacts being judged.
Feedback strength is not equal across environments, and the paper says so. Chess has the strongest in-loop feedback because a separate expert engine estimates the value lost by each chosen move. Go and Hex reflectors receive lighter deterministic diagnostics, with stronger engine adjudication reserved for a separate offline failure analysis that cannot improve the agents. NetHack records rich execution telemetry but has no optimal-action oracle. Comparisons across games therefore reflect different feedback conditions, not a single common scale.
Improvement trajectories diverge despite comparable opportunities
Across environments, frontier models show sharply different improvement trajectories even when given comparable opportunities to learn. Some achieve substantial and persistent gains on held-out interactions. Others remain near or below their initial performance. Gains made within the training regime also often transfer only partially to out-of-distribution conditions, so improvement in the training distribution is not a guarantee of broader competence.
NetHack makes the divergence concrete. Over checkpoints 0 to 40, Claude Opus 5.5 sustains higher performance after an early gain, with substantial variation across checkpoints, while the other lineages' changes are modest or negative. The paper reports a fitted total rise of 68.95 normalized points for that lineage, with checkpoint 11 the first observed checkpoint reaching 90% of that rise. Its fitted gain is 66.25 points against a measured prior learning cost of $1,072.73, giving a plasticity of 61.75 points per $1,000. No significant performance increase is observed for the other models.
Those numbers come with conditions that matter. Plasticity scores rescale mean NetHack scores using fixed anchors, 213.6 mapping to 0 and 6,526.98 mapping to 100; the anchors define units, not bounds, and the performance plots keep raw scores. The estimate also depends on score headroom, the learning horizon, saturation-fit assumptions, and provider pricing. The paper itself argues for reading full learning curves alongside any scalar plasticity value rather than treating small differences as a universal ranking.
The best final score is not the best learning rate
Endpoint capability and acquisition efficiency come apart in this study. The agent that ultimately performs best need not be the one that improves most efficiently, because a high final score can reflect a strong starting point rather than a large gain. The paper states this directly as a reason endpoint performance alone is insufficient.
The NetHack artifact comparison reinforces the point that more machinery is not the same as more learning. At checkpoint 10, where all lineages play the same ten games, Claude Opus 5.5 leads with a mean score of 9,015, while Claude Opus 5 has roughly three times as much tool code (13 files and 8,237 lines versus 6 files and 2,713 lines) yet scores 1,919. The paper also flags caveats: Claude Opus 5.5 already scored 2.6 to 25 times the others at checkpoint 0, its three best games supply 69% of its checkpoint-10 total, and other lineages used earlier harness versions. More code or more tool use does not by itself explain the lead.
Failures cluster at different stages of the improvement loop
Tracing failures through the loop reveals different candidate bottlenecks rather than one shared weakness. Agents with low plasticity often fail to reuse relevant artifacts at all. More plastic agents may still fail even when they do reuse relevant artifacts, which points toward limitations in artifact quality, generalization, or application.
The diagnostic separates three mutually exclusive failure categories: no covering artifact existed, a covering artifact existed but went unused, or a covering artifact was used and the failure remained. Because these categories condition on failure, the study separately measures decision-level reuse: among decisions where at least one saved artifact is relevant, the fraction at which any relevant artifact was materially used. Artifact relevance and failure coverage are distinct, since an artifact can be broadly relevant without addressing a particular mistake.
The paper is explicit that these diagnostics are observational. Artifact use does not establish causal effectiveness, and failure after use does not distinguish an inadequate artifact from inadequate application. Controlled interventions such as artifact removal, replacement, or forced invocation would be needed to separate those mechanisms. Concrete cases show how differently the loop can break: one agent's rules against a known early capture accumulated across checkpoints while the mistake continued, because a fixed opening book answered before the checks ran; another agent's notes told it to copy the first legal move listed, and at checkpoint 20 every held-out move was that first move, including losing ones.
Plasticity is a protocol property, not a model ranking
Plasticity is a property of an agent under a particular learning protocol, not an intrinsic scalar property of the underlying model. Long-context capability may support reflection, artifact construction, and reuse, but plasticity asks whether those mechanisms translate experience into persistent held-out gains relative to learning cost. High artifact reuse can coexist with very different task performance, so retrieval alone does not explain it. The experiments do not isolate how much long-context instruction following contributes to building or applying artifacts.
The scope has further edges. Fresh contexts and frozen weights confine persistent adaptation to self-constructed artifacts, but they do not separate the value of environment feedback from the extra computation spent analyzing experience and revising artifacts; budget-matched reflection without feedback would be needed for that. Separately trained lineages do not establish transfer of artifacts or improvement strategies across tasks. Finite evaluation sets add checkpoint-level noise.
Report the curve, then ask which stage failed
The practical priority this paper sets is a comparison rule: judge self-improving agents by held-out gains per unit of learning cost across full trajectories, and report the curves alongside any scalar plasticity figure. A single endpoint score, or a single plasticity number read as a universal ranking, cannot carry that judgment given the dependence on headroom, horizon, fit assumptions, and pricing.
The second priority is diagnostic. When an agent fails to improve, the first question is whether relevant artifacts existed and were used. Unused artifacts suggest retrieval or deployment problems; used artifacts followed by failure shift attention to artifact quality, generality, or application. The paper's own framing points further: validation of persistent artifacts should assess behavioral effectiveness, not merely whether they are structurally valid or executable, and plasticity itself could become a target for post-training rather than only a measurement.
Sources used in this answer
Agent Plasticity: Measuring Self-Improvement Through Experience
Introduces agent plasticity, the efficiency of converting experience into held-out performance gains, and shows frontier agents diverge sharply in improvement trajectories, transfer, and failure bottlenecks under a frozen-weight protocol.