Do language models report impossible engineering problems as solved?

Rejection is scored separately from solving

The paper asks a question that a normal accuracy score cannot answer: does a model that can solve a mechanics problem also reject it when one datum or premise makes the stated state impossible? Each problem comes as a pair, a consistent statement and a flawed twin that changes one given value or assumption so the stated state cannot exist. Every reply must end with a status, "solved" or "cannot solve".

Rejection is defined as a "cannot solve" status or a withheld answer. That is correct for a flawed twin and a false alarm for a consistent one. Because solving of valid problems and rejection of their flawed counterparts are scored separately, a model can score well on one and poorly on the other, and the paper reports both.

Two independent solvers certified every flaw

The main set is 30 pairs drawn from five families of multi-regime problems, which couple two regimes that textbook shortcuts treat separately, such as elastic and plastic behaviour. Examples include a settled two-span beam combining elastic response, plastic collapse and cyclic shakedown, and a beam on a bed that pushes but never pulls, where the flaw contradicts an end's uplift with a downward gauge reading.

Two independent numerical implementations of the same governing model verified every answer key, and each flawed problem was shown to be physically impossible. Acceptance gates required the two references to agree to 10^-6 relative on every numeric answer, kept the governing ratio at least 5% away from its switching value, and required every core shortcut to miss at least one numeric answer by more than 10%. OpenSeesPy provided third-party checks for three families. The initial prompts did not warn that problems could be flawed.

Twelve of ninety replies missed a flaw

Across three recent models, 12 of 90 replies failed to reject a flawed problem. In 11 of these replies, the model stated the flaw, answered a corrected problem and still reported the original as "solved", according to artificial intelligence raters and numerical checks.

The misses were concentrated. Eight of the 12 came from one family, the beam on a no-tension bed, where setting aside a gauge reading that contradicts the mechanics is a natural repair, and three came from the two-span beam with a load past collapse. On the two flaw designs of the added pairs, a rise too small to snap and a fit that the spin test loosens, these three models rejected 35 of 36 twins. Two blinded coders classed every noted reply as a repair or a continuation, agreeing on 25 of 26.

A status field that can say flawed

The paper later retested four models from one provider, replacing "cannot solve" with "flawed" and asking them to name and explain the defect. Three models showed statistically significant increases in rejection, but valid-problem solving fell in three. A score read from the status field alone therefore mixes whether a model notices a flaw with how it reports that judgment.

In the first collection, a warned arm on the original nine pairs changed two things at once: it told the model to check the statement and not to solve a flawed one, and it offered a "flawed" status with a defect type. On the same nine twins, the warning raised rejection in every model that had missed any of them, from 3 to 9 for Opus 5, 1 to 9 for Opus 4.8 and 0 to 8 for Sonnet 5, but it also added false alarms in five models, up to three of nine. Two further arms gave each part alone to five Anthropic models; either part alone gave 43 rejections of 45 flawed twins, against 44 with both and 18 with neither, with no false alarm, though the arms ran at different times.

Rejection separates models that solve alike

On this panel, solving and rejection agree broadly, with a Spearman correlation of 0.70, but among models with similar solving the second measure separates sharply. GPT-5.5 solved 16 of the 25 solve items and Opus 4.8 solved 19, yet they rejected 28 and 7 of the 30 flawed twins.

In the first collection, rejection also separated two newer Anthropic models from their older versions, Opus 5 from Opus 4.8 and Sonnet 5.5 from Sonnet 5, and in an exploratory restriction both gaps remained on pairs whose consistent twin both models solved. The retest repeated only the Sonnet gap, on 10 of the 21 added pairs and not significantly on the 7 with a reply from both models. The Opus gap shrank: Opus 5 rejected 12 flawed twins and Opus 4.8 rejected 9, against 23 and 7 before, while the consistent twins they solved barely changed. The evidence that rejection measures something beyond solving therefore rests on such differences across the panel, and no version gap that solving lacks has yet been replicated.

Misses differ by model and by family

The failure modes are not the same across models. Haiku 4.5 and gpt-oss-120b mostly show no premise check in the returned reply, and gpt-oss-120b's rejections are refusals to compute. Sonnet 5 examines the premise and concludes the wrong way. Opus 4.8 states the contradiction in 8 of its 23 misses and in 5 of those 8 returns numbers for the impossible state. The three recent models, Fable 5.1, Sonnet 5.5 and Opus 5, state it and, in all but one case, answer for a repaired reading that an engineer reading the reply can see.

A pipeline that reads only the status field sees a solved problem with numbers attached, and in five of the eleven repairs the repair appears only in the prose, not in the structured answer. Which reading matters depends on who consumes the answer; a status-level veto is what an automated workflow can act on.

Where the paired instrument does not transfer

Several limits bound these claims. The multi-regime sets are small, 25 solve items and 30 pairs from five families, too small to rank neighbours inside the top cluster, and the families were screened on Sonnet 5, so every comparison holds for these five families. The confirmatory tests are the six rejection contrasts on the 21 pairs added after the analysis plan was written; the analyses on pairs that both models solved, the rejection-reason classes and the repair split are exploratory and were defined after the replies were read. The warned-prompt result rests on nine twins and one wording per arm.

The collection and response format also limit the comparisons. The status-field comparison rests on four Anthropic models, its two arms ran over the same two periods but were not interleaved call by call, and Sonnet 5's fresh unwarned run is incomplete. Three harnesses wrap the providers with different residual system text and incomparable token accounts, access was by subscription at one effort setting on the versions served from 28 September to 1 October 2026, and a change in the served models between collections cannot be excluded. Problems were text, closed-book, with no tools and no drawing, so transfer to an agent that can call a solver is untested. The miss taxonomy and the reason and repair classes come from AI coders and an AI adjudicator, not from expert raters.

Score both twins and read the status

The paper's practical priority follows from the paired design rather than from any single accuracy number. Where a model solves the consistent problem and still returns numbers for the flawed twin, the answer format should carry a status that can say the problem is flawed, the prompt should ask for the premise check, or a verifier in the loop should be able to refuse the answer, and both twins should be re-measured whenever the model, its service or the format changes, because a format that raises rejection can also lower solving.

The most decision-relevant split is between flaw recognition and the reported status. The three recent models noticed the flaw in 11 of their 12 missed rejections and still reported "solved", and in five of those eleven repairs the repair appeared only in the prose. A reader or pipeline that trusts the status field alone would treat an impossible problem as solved with numbers attached, which is exactly the failure the twin is built to expose.

Sources used in this answer