Do biological reasoning models actually use their biological inputs?

Accuracy does not prove the sequence was read

Biological reasoning models connect a language model to biological foundation model representations and to biological text, then are post-trained on tasks such as disease prediction, protein function annotation and cell type annotation. Their benchmark accuracy is commonly read as evidence that the language model reasons over those inputs. This paper tests that reading directly rather than inferring input use from scores.

The difficulty is that a model usually receives several representations of the same genome, protein or cell at once. Text annotations can support a correct answer even when the accompanying sequence or transcriptomic representation is ignored, and representations can encode overlapping biological properties. A small performance contribution from one input could therefore mean the input is uninformative, that another input already supplies the answer, or that the model simply does not use it. Benchmark accuracy alone cannot separate these cases, which is why the study builds separate tests for each.

Four tests separate encoding from use

The authors apply four complementary probes to each model. In a perturbation test they change one biological input while holding the query and other inputs fixed, then compare accuracy against the intact version. In an evidence conflict they pair the foundation model representation of one entity with the text of a different entity and record which source the prediction follows. A linear probe asks whether the representation the language model receives already contains the task target, which shows encoding but not use. Finally, the generated reasoning traces are checked against the actual biological input.

These tests answer different questions and the paper keeps them apart. A probe that predicts the target from a representation shows the information is present; it does not show the language model exploits it. A perturbation that leaves accuracy unchanged shows the input is not needed for that task, but only if the input carried predictive information in the first place. The evidence-conflict design supplies the sharpest contrast because both candidate answers remain available and the model must effectively choose a source.

BioReason follows text over DNA

BioReason receives projected Evo2 representations of a reference and a variant DNA sequence alongside chromosome, gene, KEGG pathway and network text, and is scored on disease prediction over 1,449 KEGG-derived queries covering 708 genomes and 37 disease labels. Shuffling both DNA windows left accuracy essentially unchanged: 0.847 intact versus 0.847 shuffled for the RL checkpoint, and 0.823 versus 0.814 for the SFT checkpoint, a net change of +0.001 and −0.009 respectively.

The evidence conflicts make the same point more directly. When the DNA representation came from one genome and the pathway or gene text from another, BioReason followed the text in 97.9% of cases. A linear probe trained on the Evo2 representations could still predict the task targets, so the encoder does carry task-relevant information. The gap between that finding and the near-zero perturbation effect is the paper's central distinction: the representation is informative, but the post-trained model does not depend on it.

BioReason-Pro rarely needs its protein encoder

BioReason-Pro combines ESM3 sequence representations with GO-GPT and InterPro text for protein function annotation. Permuting the amino acid sequence changed the emitted GO term set for 6,593 of 14,102 proteins, or 46.8%, but improved F1 for 11.2% and worsened it for 10.8%, leaving 78.0% unchanged; the mean Jaccard overlap of emitted term sets was 0.889. In evidence conflicts pairing one protein's ESM3 representation with another's text, the model followed the text in 99.7% of cases.

The authors also report what the traces do under these conflicts. BioReason-Pro traces describe functions that are omitted from the final prediction, so the stated reasoning does not reliably indicate which input drove the answer. As with BioReason, the encoder is not empty of signal; the point is that the post-training procedure does not make the final prediction depend on it.

Some models do depend on their biological inputs

The result is not uniform across the six models. Foundation model inputs contribute to the performance of ChatNT, Prot2Text-V2 and CellWhisperer, and for Cell2Sentence-Scale the differentially expressed genes in the gene sentence contribute to performance. The paper's claim is therefore about post-training practice in specific models, not an inherent property of biological reasoning models.

This contrast matters for how the finding should be read. The same evaluation framework produces both outcomes, so the difference is attributable to the model and its training rather than to the test. A reader comparing architectures should ask which input the model was actually trained to rely on, not simply whether a foundation model representation was supplied.

Higher accuracy does not mean more input use

Across SFT and RL checkpoints of BioReason-Pro and 42 BioReason checkpoints, increases in accuracy did not imply greater performance contributions from biological inputs. This is the paper's most consequential comparison because it undercuts the natural assumption that a better-scoring model has learned to read its sequence or structure input more effectively.

The authors connect this to the training signal. Post-training rewards correct final answers, and when accompanying text already supports the target, a correct answer can be reached without extracting complementary information from the representation. The paper's own framing is that current post-training strategies do not ensure that foundation model representations contribute to task performance, and it proposes rewarding correct responses to changes in biological inputs as a direction to evaluate.

Where these conclusions stop

The study evaluates six models on specific tasks and biological inputs, so the findings may not generalize to other architectures or modalities. Some analyses shuffle sequences, which can push representations outside the distribution of natural sequences; the authors note that evidence conflicts built from unmodified inputs support the same conclusions. Linear probes show that a target is predictable from a representation, but not how the language model could use it.

Within those bounds, the practical priority is specific: for a model like BioReason or BioReason-Pro, a high benchmark score is not evidence that the DNA or protein encoder is doing work, and the perturbation and conflict tests are what distinguish the two. The paper's closing recommendation follows from that split rather than from accuracy alone, asking designers to treat correct use of task-relevant biological inputs as an objective alongside predictive accuracy.

Sources used in this answer