Three scores for one build
BrickBench asks an agent to turn a text prompt into a LEGO assembly that is both faithful to the description and physically buildable. The benchmark scores each result on three separate axes, and keeping them apart is essential to reading the results. Validity checks whether the assembly can exist: parts must connect, must not collide, and must stay at rest under gravity in simulation. Semantic alignment asks whether the build depicts the prompt, measured as the share of yes/no questions answered correctly after the prompt is decomposed into a question graph. Design quality is a pairwise preference score, reported as ELO, for how well the assembly is designed rather than merely whether it works.
The three axes answer different questions, so a high score on one does not imply a high score on another. An assembly can be valid and faithful yet still be a poor design, which is exactly the gap the benchmark was built to expose.
Scale and inventory change the task
The benchmark contains 300 prompts, split into three settings of 100 prompts each, with ten prompts in each of ten topic categories. In the Model setting an agent may use at most 400 parts. The Set setting requires 400 to 4000 parts, chosen to mirror the scale of the BrickLink Designer Program. The Alt-Build setting restricts the agent to the 783 pieces and 104 part types of one retail set, testing design under a scarce fixed inventory. Both Model and Set draw on the full LDraw library of 14,441 part types.
These settings are not interchangeable. Model tests assemblies that are mostly a single connected object, Set tests full retail-set scale, and Alt-Build tests whether an agent can work within a limited parts bin. An assembly that breaks a setting's part limits fails the validity check, so the setting also defines what counts as a legal answer.
Tools, not instructions, drive buildability
BrickAgent is the environment in which agents build. It provides the LDraw library, the connector system, construction tools, and interpretable validation of connectivity, collisions, and stability. The design choice is deliberate: the environment supplies building blocks, constraints, and feedback, but does not prescribe how to design the assembly, leaving the agent to devise its own solution.
The ablation shows how much this matters. With the same agent command-line interface, container, and task requirements, but only the LDraw library and a primer on the file format in place of the tools, GPT-6 Astra remained valid on 40% of prompts while GPT-5.6 Luna produced only one valid build across 300 prompts. Their assemblies averaged 7 and 156 colliding pairs respectively, and split into 6 and 83 connected components, against 2 and 5 with the environment. Reliable construction therefore depends on grounded tools and feedback rather than on the model alone.
Validity is nearly solved inside the environment
Across the eleven frontier agents evaluated, validity is close to saturated when the environment is available. Five agents delivered a valid assembly for every prompt, and none fell below a validity proportion of 0.80 overall. Of the 137 invalid assemblies in the reference set, 44 were prompts with no delivered assembly at all; the rest failed on stability (41), collisions (31), and the part requirement (21).
The settings differ in what breaks. Set is the hardest for validity, dropping to 0.68 for Muse Spark 1.3, 0.79 for Qwen 3.8 Flash, and 0.87 for DeepSeek V4.1 Flash. Alt-Build costs the least and keeps the highest validity. The two data-driven baselines fall far below the agents: BrickNet-14B is collision-free on only 26% of prompts, and BrickGPT, though valid on each prompt, satisfies 2% of the questions.
Prompt adherence saturates before design does
Semantic alignment separates the agents only gently, from 0.59 for GLM 5.3 Flash to 0.95 for GPT-6 Astra. Design ELO separates them far more sharply: the 545-point gap between GLM and Astra corresponds to a judge preferring Astra's assembly in roughly 96% of pairs. Within the reference set Astra leads on every metric, and Claude Opus 5 ranks second while incurring four times the cost. GPT-6.1 Sol matches Astra on ELO, 1293 against 1297 with overlapping intervals, at a fifth of the cost.
Alignment also varies more by question type than by subject. Every agent satisfies at least 91% of the questions about color, but texture is satisfied only 49% to 83% of the time, with object type and state the next hardest. No prompt category is consistently harder than the others, and the two top agents score at least 0.90 in every category. The stronger agents also build larger and more varied assemblies: in Model, Astra averages 217 parts of 35 types in 9 colors, while GLM averages 27 parts of 10 types in 5 colors.
Cost does not track design rank
The reference set spans a factor of 30 in cost, and rank does not follow price. Muse Spark 1.3 costs nine times what DeepSeek V4.1 Flash costs and rates 138 ELO points lower. Alignment and design scores also diverge for some agents: GPT-5.6 Sol scores 53 points higher on alignment than on design. Cost remains an open axis, since BrickNet-14B produces an assembly for six hundredths of a cent while Astra does so in $4.06.
This matters for anyone choosing a system rather than ranking one. A cheaper agent can land in the middle of the ELO range, and the most expensive option is not automatically the best design. The paper's own comparison of GPT-6.1 Sol and Astra, near-equal ELO at very different cost, is the clearest example of that split.
Human designs remain distinguishable
To test design quality against people, the authors asked raters familiar with LEGO design to tell agent builds from human ones. For Model and Set assemblies, they drew part-count-matched human-designed models from the BrickNet dataset and asked which of the two a person made. Five raters provided 360 ratings and selected the human model in 323 of them. No agent was taken for a human designer more than one time in five, and every agent's rate fell significantly below chance; GPT-6 Astra, the agent most often taken for a person, was chosen in 4 of its 21 pairs.
The comparison has a stated boundary: the human samples are matched on part count but not on prompt, because no human designs exist for the BrickBench prompts. A prompt-matched comparison would require commissioning designs. Within that limit, the authors interpret the result as a measure of design quality: agents satisfy requirements but fall short of the standards of a designed set, lacking the part usage, proportion, and surface detail that mark human design.
The stability check is necessary but not sufficient
The physical validity score comes from a simulator, not from building the model. Connected components are treated as rigid bodies of uniform density in PyBullet, with high friction and no restitution, and an assembly is stable if no point of any component moves more than 3 LDraw Units, three quarters the height of a brick stud. Effects such as the holding force of stud connections and the play of axles are not modeled, so an assembly that balances as a rigid body passes even when a built one might sag or fall apart.
The authors are explicit that this bar is necessary but not sufficient, and that testing connection strength would require a structural simulation with force limits or a finite-element model, which would severely limit the available part catalog. The design judge has its own validation: against human pairwise judgments from 102 raters and 4,080 judgments, the ELO rankings agreed on 34 of 36 agent pairs, with a Kendall tau of 0.89.
The remaining gap is design, not compliance
The paper's four findings point to one priority. Data-driven models built specifically for the task are subsumed by general coding agents, which exceed them on their own benchmark and on BrickBench. Prompt adherence has saturated, with the best agent satisfying 95% of the VQA questions and absolute alignment no longer strongly separating top performers. Performance depends on the environment, since the same model produces buildable assemblies with BrickAgent and unbuildable ones without it. And a gap to human design remains, with raters filtering agent builds from human ones nine times out of ten.
For readers comparing systems, the decision-relevant split is therefore not validity, where five agents already reach 1.00, but design ELO and its cost. Astra leads on every metric, yet GPT-6.1 Sol matches its ELO at a fifth of the cost, and Opus 5.5 rates 148 points above Opus 5 at 40% of its cost while sitting 83 points lower on alignment. The benchmark's own conclusion is that agents are learning to build what works, not yet what makes a great design.
Sources used in this answer
BrickBench: Evaluating Agentic Brick Design
Leading coding agents produce physically valid and largely prompt-faithful LEGO assemblies inside the BrickAgent environment, but human raters identified human-made models in 323 of 360 comparisons, showing design quality remains the main gap.