BAGEL: Testing the Limits of LLM Knowledge in the Animal Kingdom

TL;DR

LLMs are increasingly used for scientific synthesis, but how much do they actually "know" about the natural world? BAGEL (Benchmark for Animal knowledGe Expertise in Language models) is a new 11.8k-item benchmark that reveals a significant gap between general language proficiency and specialized biodiversity knowledge. While frontier models handle Wikipedia facts well, they largely fail at complex ecological interactions and bioacoustic reasoning.

The "Generalist" Blind Spot

Current LLM evaluation relies heavily on benchmarks like MMLU. While these capture broad scientific literacy, they often overlook the "long-tail" of natural history. For ecology and biodiversity conservation, we need models that understand specific species interactions (who eats whom?), scientific results from preprints, and the nuances of animal communication.

The authors' central insight is that animal expertise is heterogeneous. A model that knows a cheetah is fast (encyclopedic) might not understand the specific frequency modulation of a Red Warbler's song (bioacoustic) or the parasitic relationship hidden in obscure biotic interaction databases.

Methodology: A Four-Pronged Approach

BAGEL doesn't just scrape any data; it targets four distinct "anchors" of knowledge using specialized pipelines:

1. Wikipedia: Encyclopedic recall across dimensions like Diet, Habit, and Morphology.
2. GloBI (Global Biotic Interactions): Reasoning about ecological links (e.g., host-parasite relations).
3. bioRxiv: Interpreting experimental results from real scientific preprints.
4. Xeno-canto: Testing textual knowledge of animal sounds (frequency, duration, modulation).

Figure 1: The curation workflow showing how different source tracks are unified into a four-option MCQ format.

One of the most rigorous aspects of this study is the mitigation of answer-position bias. Early in the study, the authors found that models were exploiting the fact that correct answers were often in the same letter position across certain domains. They solved this by applying random permutations to all choices, ensuring the results reflect actual knowledge rather than "shortcut" learning.

Experimental Insights: The Bioacoustic Barrier

The results from the leaderboard offer a sobering look at current SOTA models.

Table: Accuracy of Frontier and Open-weight models on BAGEL.

Key Observations:

• The Fragility of Scale: In the Qwen3 family, increasing parameters from 14B to 32B actually decreased accuracy on Xeno-canto tasks. This suggests that as models become more "general," they might inadvertently lose or fail to capture highly specialized lexical registers like bioacoustic jargon.
• Proprietary vs. Open: There is a roughly 10% gap in overall accuracy between the best open-weight model (Gemma 3 27B IT) and GPT-5.4.
• Vocabulary vs. Reasoning: The authors analyzed the "Zipf" scores (word frequency) of the question stems. Xeno-canto questions contain significantly more rare tokens and bio-specific terminology, which correlates with the high failure rate in that domain.

Critical Analysis: Is "Closed-Book" Enough?

The study highlights that performance on textual descriptions of sounds is consistently the lowest across all models. This raises a fundamental hardware/software question: Should a text-only LLM be expected to understand the harmonic structure of a bird call? The authors argue yes—not because the model is "hearing," but because scientific communication relies on a shared textual vocabulary for these phenomena. If a model can't reason about the term "sinusoidal vibrato" in a text-based research context, its utility for ecologists is limited.

Conclusion

BAGEL provides a necessary reality check for the field. It proves that:

1. Animal knowledge is not a monolith: Factual recall (Wiki) does not equal reasoning (bioRxiv/GloBI).
2. Positional bias is rampant: Evaluators must shuffle options to avoid overestimating model intelligence.
3. Specialization is needed: The failure of even the largest models on bioacoustic and interaction reasoning suggests we need models specifically pre-trained on the "Tree of Life."

As we move toward AI-driven biodiversity monitoring, benchmarks like BAGEL will be the gatekeepers for ensuring these models don't just hallucinate natural history, but actually understand it.