Models keep reading after they already know
Large language models process every token in an input context with the same computational priority, regardless of whether that token matters for the question. When the answer sits in the first few sentences of a long document, the model still processes the rest. The paper frames this as a fundamental inefficiency and links it to the "lost-in-the-middle" problem, where critical information gets diluted in long inputs.
The authors contrast this with human cognition: people assess whether they have enough evidence and stop gathering once they do. Their question is whether a model can make the same judgment internally, without an externally imposed compression target. That framing matters because it changes who decides how much context is needed — the model's own internal state rather than a fixed ratio chosen in advance.
Sufficiency signals concentrate in a few attention heads
The method rests on an analysis of model internals. The authors probed attention heads across all layers and found that a subset, primarily from middle layers, predicted context sufficiency far better than others. They select only the top five heads by validation F1 to train an ensemble classifier, and report that adding more heads yields minimal gains or even worse performance.
That concentration is the paper's key mechanistic claim: sufficiency information is not spread evenly through the network but sits in specific architectural components. The selected layers vary by model architecture; the authors let probing results, not a predetermined layer list, decide which heads to monitor. A lightweight classifier then reads those activations and decides when processing can stop.
Dynamic cutoff differs from fixed compression targets
Existing context-efficiency methods typically predefine a target. The LLMLingua family uses a small language model to filter unimportant tokens down to a fixed compression rate, and retrieval-augmented generation fixes the number of top-k documents. The paper groups both as static methods because every input receives a uniform reduction — 50% compression means every input is halved, whatever its content.
Dynamic context cutoff instead adapts per input: different inputs receive different amounts of compression based on the model's own assessment. The authors emphasize that their method requires no predefined compression target and no model retraining. This is the comparison that gives the headline numbers meaning — the 1.33x reduction is not achieved by applying the same ratio everywhere.
Accuracy rises while token processing falls
The evaluation covers six QA datasets with contexts up to 40K tokens, spanning single-hop tasks (code understanding, SQuAD, Natural Questions), synthetic multi-hop key-value retrieval, and natural multi-hop reasoning (HotpotQA, MUSIQUE). Three model families were tested — LLaMA, Qwen, and Mistral — across sizes from 1B to 70B parameters.
On this setup the method reports a 3.4% accuracy improvement together with 1.33x average token reduction. The paper also states that at equivalent token reduction rates, the method outperforms other context-efficiency approaches. Both figures are averages over the described benchmark suite; they are not a guarantee for any single dataset or model size.
The dataset design is worth noting when reading these numbers. Gold information locations were sampled from a uniform distribution, so roughly half of context chunks are labeled insufficient and half sufficient. That balance keeps the evaluation from favoring early or late stopping decisions, but it also means the benchmark deliberately places answers throughout the context rather than at the start.
Larger models can judge sufficiency without probes
The paper reports an emergent scaling phenomenon. Smaller models require probing — the trained classifier on selected attention heads — to detect sufficiency. Larger models instead exhibit intrinsic self-assessment capabilities through prompting, meaning they can be asked directly whether they have enough information.
This splits the practical picture by model size. For smaller models, the method is a probing pipeline: identify informative heads, train a classifier, and use its output as a stopping signal. For larger models, the same judgment can come from the model itself. The authors present this as an observed scaling pattern rather than a designed feature, and it is the paper's most consequential claim for anyone deciding how to deploy the method.
The stopping rule depends on a universal cutoff assumption
The authors' own limitations discussion centers on assumptions made for a universal cutoff point, and they leave exploration of those limitations to future work. The sufficiency classifiers are also trained on labeled context chunks; when explicit answer locations are unavailable, the paper tests synthetic labels generated by GPT-4o. Those classifiers still perform competitively but trail human-annotated labels — overall F1 of 84.6 versus 88.3 at 1B and 87.0 versus 89.8 at 8B — so the labeling source measurably shifts detection quality.
The benchmark scope also bounds the claims. Results come from QA datasets with gold locations sampled uniformly, contexts up to 40K tokens, and three model families. The paper does not establish that the same stopping behavior holds for tasks without a localized answer span, and the balanced sufficient/insufficient split is a property of the evaluation design rather than of natural workloads.
Model size decides which sufficiency mechanism to use
The paper's most useful distinction is not the average speedup but the split between probing and prompting. If you are working with a smaller model in the tested families, the evidence supports building a probe over a handful of middle-layer attention heads — five sufficed here, and more did not help. If you are working with a larger model, the paper's scaling observation suggests asking the model to assess its own sufficiency may substitute for that pipeline.
Between those paths, the reported trade-off is consistent: dynamic, per-input stopping beat static compression at matched token reduction, and it did so while raising accuracy rather than sacrificing it. The number to carry forward is the pairing — 1.33x fewer tokens processed and 3.4% higher accuracy on the described QA suite — with the caveat that the stopping decision rests on a cutoff assumption the authors flag as unresolved.
Sources used in this answer
Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signals
Dynamic context cutoff uses lightweight classifiers over a few middle-layer attention heads to detect when a model has enough information, achieving 1.33x average token reduction with a 3.4% accuracy improvement across six QA datasets and three model families, while larger models can self-assess sufficiency through prompting.