Why can't we just use one benchmark for all long-context tasks?
The fundamental reason is that current LLMs are surprisingly bad at handling long contexts, and the specific ways they fail differ by task type. A 2024 study [4] evaluated models on LooGLE, a benchmark with documents over 24,000 tokens, and found that most LLMs have "shockingly bad long context ability" — they fail to capture long dependencies even when their context window is large enough to fit the entire document. This means a model might ace a short reasoning question but completely miss a key fact buried 20,000 tokens into a story. If you used the same benchmark for coding, reasoning, and creative writing, you'd miss these task-specific failures.
A 2025 study [5] on long code understanding (LongCodeU) found that LLM performance drops dramatically when code length exceeds 32,000 tokens, falling far short of their claimed 128K to 1M token context windows. The most challenging aspect was "inter-code unit relation understanding" — essentially, connecting pieces of code across a large repository. This is a very different skill from, say, summarizing a long novel or answering a question about a scientific paper. A single benchmark would lump these together, hiding the fact that a model might be decent at long-document QA but terrible at long-code reasoning.
What do current benchmarks tell us about the differences between task types?
Each benchmark highlights unique challenges. For long-context understanding in general, the 2024 LongBench [1] — the first bilingual, multi-task benchmark — tested 8 LLMs across 21 datasets and found that even the best commercial model (GPT-3.5-Turbo-16k) struggled on longer contexts. Context compression techniques like retrieval helped weaker models but still lagged behind models with strong long-context ability. This shows that "long-context" is not a single skill; it's a bundle of skills that vary by task.
For coding specifically, the 2025 YABLoCo benchmark [2] focused on generating function bodies in large C/C++ repositories (200K to 2,000K lines of code). This is a radically different challenge from, say, answering a question about a 50-page document. The benchmark includes call graphs and dependency contexts — things irrelevant to creative writing or reasoning. Similarly, the 2025 LongCodeU benchmark [5] found that inter-code unit relation understanding was the hardest aspect, a task that simply doesn't exist in most long-document QA benchmarks.
For reasoning and information retrieval in long contexts, the 2025 Russian-language benchmark [3] categorized tasks into four levels of complexity up to 128K tokens, covering information retrieval, knowledge extraction, and reasoning. This shows that even within "long-context," the difficulty scales with context length and task type. A model that can retrieve a fact from a 128K document might still fail to reason across multiple sections of that same document.
So what should benchmark designers do instead?
The evidence strongly suggests that benchmarks should be task-specific. The 2024 LooGLE study [4] explicitly calls for enhancing "true long-context understanding" rather than just enlarging context windows — and that requires benchmarks that test long dependencies, not just short-range recall. The 2025 LongCodeU study [5] echoes this, noting that models' claimed context windows (128K to 1M tokens) are misleading because performance collapses well before those limits for code tasks.
In practice, this means: use a dedicated coding benchmark (like YABLoCo [2] or LongCodeU [5]) for code generation and understanding; use a long-document QA benchmark (like LongBench [1] or LooGLE [4]) for reading comprehension and reasoning; and design separate tests for creative writing (which would need to evaluate coherence, plot consistency, and style over long texts — aspects none of these benchmarks cover). A single benchmark would be too coarse to reveal these critical differences, and would likely overestimate or underestimate model capabilities depending on the task mix.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2024 to 2025, 5 from 2024 or later, collectively cited 98 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 34 papers retrieved from a database of over 500 million.
Sources used in this answer
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
LongBench, the first bilingual multi-task benchmark for long-context understanding, evaluated 8 LLMs across 21 datasets (average 6,711 English words) and found that even GPT-3.5-Turbo-16k struggled on longer contexts, while context compression like retrieval helped weaker models but still lagged behind strong long-context models.
YABLoCo: Yet Another Benchmark for Long Context Code Generation
YABLoCo introduced a long-context code generation benchmark for C/C++ with 215 functions from repositories of 200K to 2,000K lines of code, providing a scalable evaluation pipeline for function body generation in large repositories.
Long Context Benchmark for the Russian Language
A Russian-language long-context benchmark with 18 datasets across four complexity levels (up to 128K tokens) was created to evaluate LLMs on information retrieval, knowledge extraction, machine reading, question answering, and reasoning.
LooGLE: Can Long-Context Language Models Understand Long Contexts?
LooGLE, a long-context benchmark with documents over 24,000 tokens and 6,000 questions, found that most LLMs have 'shockingly bad' long-context ability, failing to capture long dependencies even when their context window is large enough.
Benchmarking Long-Context Language Models on Long Code Understanding
LongCodeU evaluated 9 LLMs on long code understanding across 8 tasks and found performance drops dramatically beyond 32K tokens, far short of claimed 128K to 1M context windows, with inter-code unit relation understanding being the most challenging aspect.