Can Models Be Evaluated by the Robots They Engineer in a Simulated Physical Arena?

The arena scores built artifacts, not answers

ArtifactArena replaces text-based evaluation with a physically grounded hardware-software co-design challenge. A model must engineer a fully functional robot, meaning it writes both the morphology and the controller, and that robot then competes in a simulated arena. The paper frames the motivation through Betty, a New Caledonian crow that bent a straight wire into a hook to retrieve food in 2002: what evidenced her intelligence was the physical artifact, not anything she said.

This matters because the unit of evaluation changes. The score is not a benchmark accuracy on a question set; it is the competitive performance of a submitted bot artifact against other bots. A model's capability is therefore read off a design that must compile, satisfy constraints, and survive matches, rather than off a generated answer.

Three harnesses separate sampling from refinement

The paper evaluates a frontier model's zero-shot, verifier-guided refinement, and open-ended physical design capabilities through three harnesses. Sampling (SH) is the zero-shot condition. Verifier-Grounded Refinement (VGH) refines bot artifacts using physics simulator feedback. Design Lab (DLH) is the open-ended physical design condition, where refinement is driven by text descriptions and gameplay data.

These are not interchangeable measures of the same skill. In the all-harness champion matrix, each champion's harness is labeled, and the same model can appear under different harnesses with different Elo. The paper's own Figure 8 separates results by harness precisely because pooling them would conflate zero-shot generation with iterative, feedback-driven design.

Elo comes from artifact tournaments, not model debates

Ranking is derived from head-to-head tournaments between artifacts. In the champion round, each pairing pits cell champions, defined as the best artifact of each model under each harness, against each other. A cell score is (wins + half of draws) divided by games, and the mirrored cell is one minus that value; self-pairings are never played.

Selection inside a run is also artifact-based rather than draft-based. After the last turn, every bot saved with save_bot enters a round robin of three seeds in each direction, draws count as half a win, and ranking uses Bradley–Terry strength. The winner is the run's artifact, and the paper notes an earlier candidate may beat the final draft. A manual submission, an unsaved draft, or files written to an output folder cannot override this selection.

Fixed materials and rules make mass deterministic

Comparability depends on a fixed material library. Each geometry names a material, and during validation the simulator computes geometry mass from volume and material density, then overwrites friction with that material's coefficient. The palette runs from foam at 200 kg/m3 and friction 0.50 to tungsten at 19,300 kg/m3 and friction 0.40, with rubber at 1.60 friction. User-specified mass, density, body mass, and inertial tags are not used for final physics, and a geometry without a valid material defaults to foam.

Structural rules further constrain what counts as a valid design. A robot must have exactly one free joint on its root body, so no part may detach during a match; only motor actuators are permitted; each unit of gear magnitude adds 0.01 kg of actuator mass to the mounting body. The bounding box is measured on the initial pose and re-checked in the robot's own root frame at every control step, so a robot whose extents exceed the box at any step loses the match. Controllers run in a restricted Python sandbox with only math, numpy, random, typing, and collections available, and a controller exception or malformed return counts as a loss.

GPT-6 Astra leads the all-harness champion round

In the all-harness champion round, GPT-6 Astra holds the top Elo at 1539, followed by Claude Fable 5.1 at 1310, Gemini 3.8 Flash at 1223, GPT-5.3 Codex at 1173, and Kimi K3 at 1164. The same model can occupy multiple rows because each row is a champion under a specific harness, which is why GPT-6 Astra and Claude Fable 5.1 appear repeatedly near the top.

Restricting to a single harness changes the ordering. Under Sampling, GPT-6 Astra leads at 1352, ahead of Claude Fable 5.1 at 1260 and GPT-5.4 at 1179. Under Verifier-Grounded Refinement, GPT-6 Astra leads at 1539, ahead of Claude Fable 5.1 at 1310 and Gemini 3.8 Flash at 1223. Under Design Lab, the leader changes: Claude Fable 5.1 tops the list at 1210, ahead of Claude Fable 5 at 1189 and GPT-6 Astra at 1188. The Design Lab result is the clearest evidence that open-ended design is not simply a continuation of zero-shot sampling strength.

Most sampled failures are self-inflicted, not judged

Implementation failures were common among sampled robots. Of 216 robots that built but did not qualify, 125 drove themselves off the platform, 60 triggered the inactivity rule, and 29 exceeded size limits during motion. These are mechanical and control failures detected by the arena, not subjective judgments about design quality.

The revision examples show the same failure modes being diagnosed and fixed. Kimi K3 attributed its wedge's poor turning to wheel scrub torque exceeding drive torque, and revision 6, Fulcrum, placed the drive weight on two wheels at the center of mass so it could pivot in place while preserving the wedge-pushing strategy. Claude Fable 5.1 initially lost two of three rounds to size violations when its lifter rose, inferred that the size check used orientation-dependent bounding boxes, and repositioned the wheels and ramp, qualifying 3-0 from revision 2 onward. GPT-5.4 replaced a successful pusher's fixed tungsten slab with sliding ballast so the controller could shift mass toward the active pushing end or toward the platform center near an edge.

Wheel-supported bodies dominate the sampled population

The paper classifies sampled bots with a dichotomous key of 20 questions, from whether a bot has a wedge or ramp to whether a ring-like structure surrounds the chassis. The largest single class is a wheel-supported body plan with 99 bots, and the key's question index shows how many designs share each structural feature.

This classification is descriptive of the sampled population, not a performance ranking. It shows that many sampled designs converge on wheel-supported bodies, while the Elo tables show which artifacts actually won matches. Reading the key as a quality signal would confuse design frequency with competitive outcome.

The Elo ranking stops at the simulation boundary

The paper states its main limitation directly: robots are built and judged inside MuJoCo, so constructing a robot here does not mean the robot will work in the real world. The authors anticipate that as simulations approach reality, the same arena and harnesses could carry over unchanged, with the robot becoming something that can be printed, assembled, and fought for real. Until then, results should be interpreted as measuring competitive physical design and control of frontier models in simulation rather than direct real-world robotic capability.

The practical priority follows from that boundary plus the harness split. If the question is which model is strongest at zero-shot robot generation, the Sampling ranking is the relevant one, and GPT-6 Astra leads at 1352. If the question is which model benefits most from open-ended, feedback-driven design, the Design Lab ranking is the relevant one, and Claude Fable 5.1 leads at 1210 while GPT-6 Astra falls to 1188. The most decision-relevant reading of this paper is that harness choice, not a single overall score, determines which model looks best, and that every one of those rankings is a simulation result.

Sources used in this answer