Why does rehearsing everything in a KV cache remember nothing under tight budgets?

Full-context rehearsal collapses as the budget tightens

The setting is a cache compressed once, before anyone knows what will be asked of it: a document cached for retrieval, a prompt prefix shared across requests, a long conversation kept for turns that have not happened yet. Because memory grows with the number of caches, compression only pays off when it is extreme — a 5% keep ratio fits 20 times as many caches in the same memory, a 30% ratio only about 3 times. The paper targets the 3–5% regime.

The prevailing approach, rehearsal scoring introduced by KVzip, has the model reread the context while attending to the cache, then keeps the entries that draw the most attention. Its successors refine the scorer or what a retained subset can express, but all rehearse the same target: the entire context. The paper's central claim is that this shared default fails exactly where compression matters. On RULER-4K with Qwen3-8B, the uncompressed score is 96.5; full-context rehearsal at a 3% keep ratio keeps 31.5. On LongBench's natural-text tasks it scores 28.6, below methods that rehearse nothing at all, which reach up to 36.5.

The failure is not a capacity problem. A 5% cache scored by rehearsing the answer loses almost nothing — 93.5 against 94.7 uncompressed. The budget is being spent on the wrong positions, not on too few of them.

A cache keeps what its rehearsal reads

To locate the loss, the paper measures how much of each question's answer survives compression. At a 5% keep ratio, full-context rehearsal keeps 5.7–7.1% of the answer's KV entries on seven of eight needle tasks — no more than it keeps of any other token. The answer's own entries survive at roughly chance.

The mechanism the paper proposes is that a cache keeps what it rehearses. Rehearsed positions cover about 5% of the context yet draw a fifth to a third of the retained budget. When the model rehearses only a list of facts, its score on each task tracks how much of that list the rehearsal covers. Rehearsing 5% of a page takes roughly 30% of the cache, and the cache answers what the rehearsal covers.

A sharper version of the finding appears in the appendix: a cache keeps what its rehearsal reads. When the rehearsal is coherent text, the model continues the stream from its own local context and looks back into the document only for tokens it cannot guess — needle values, entities, numbers. Those are exactly what questions ask for, and they are kept. Predictable structure such as sequentially numbered passage labels draws no attention back to the document and is lost.

Rehearsing the question is not rehearsing the answer

Two controls separate what a rehearsal must contain. Rehearsing only the needle sentences — every fact that can be asked — under the same budget as KVzip+ scores 98.5–100 on every fact-sparse RULER-4K task at both 5% and 3.125% keep ratios; its lead over KVzip+ grows from 24–58 points to 74–83 points, with KVzip+ falling from 44.4 to 18.5 on multi-key-1.

The second control shows the answer matters, not just the question. Rehearsing the question alone, without its answer, scores 47.0 against 93.5 with the answer under the same scorer. While reading a question, attention lands on the key it names; only while producing the answer does it land on the value. A rehearsal must reproduce the read-out, not the request.

A deployable rehearsal cannot see the questions, but the model can anticipate them. Asked to write the questions a reader is likely to ask and to answer each by quoting the document, it produces read-outs of the spans it expects to be asked. On LongBench, an answer-aware rehearsal keeps a 5% subset that matches the uncompressed model, 44.9 against 44.4.

Attic quizzes the model and adapts the amount

Attic-KV, named after Holmes's brain-attic, builds its rehearsal from three parts. Self-generated QA rehearsal has the model write the questions it expects and answer them by quoting the context. Anchors decide what else to rehearse: within a 4K-token chunk, the top keys that stand out by being far from the average key. The length of the sentences containing those anchors decides how much to rehearse.

That third part implements the paper's second principle, rehearse as much as there is. Documents differ in how much of them can be asked: a narrative may hide one fact in a page of prose, while a table is facts from top to bottom. At an equal average amount on the RULER-4K development split with Qwen3-8B, KV2 with a fixed 35% rehearses 35% of every document and scores 74.6; Attic's content-adaptive amount rehearses 11–100% per document, 34% on average, and scores 78.0. The gain comes where content is dense.

Attic is training-free and changes only the rehearsal, so it plugs into any rehearsal-scoring host without altering its scoring, and it accepts RestoreKV's released restore tokens without retraining. Under these principles, prior methods become special cases: KVzip fixes the amount at the whole context, KV2 at a fraction, and a content-adaptive rehearsal approaches KVzip as the budget loosens.

The same rehearsal lifts three different scorers

Because Attic changes only the rehearsal, the paper can test it inside hosts that score entries in three different ways: by attention, by gradient attribution, and with trained restore tokens. Alone, Attic is the best training-free method in all eight settings tested on RULER and LongBench's natural-text tasks. At a 3% keep ratio on RULER-4K it scores 73.4, 6.4 points above KV2 and 41.9 above KVzip+.

Plugged into the gradient-based KVgrad, it raises the host by up to 17.1 points, and KVgrad with Attic beats every existing training-free method in all eight settings. Plugged into the trained RestoreKV+, it raises the host by up to 28.1 points and beats every existing method in all six settings where restore tokens exist.

The margin widens as the budget tightens, which is the regime the paper argues matters. On LooGLE, where each document is compressed once and then answers all of its questions, Attic with restore tokens leads on both short- and long-dependency questions.

Partial rehearsal also compresses faster on long documents

Because it rehearses only part of each context, Attic's compression time grows more slowly with length than full rehearsal's: it matches KVzip+ on a 32K-token context and takes 27% less on a 64K-token one. That matters for the one-shot, many-query setting, where compression cost is paid once per cached document.

Coherence carries a cost the paper documents rather than hides. On LongBench passage retrieval, whose answer is a "Paragraph N" label, the content-adaptive amount rehearses about half the context because every sentence is an askable fact. As rehearsal grows from isolated anchors into coherent text, the kept fraction of label tokens falls from about five times the 5% base rate to the base rate itself, and the score falls with it: 77 to 40 on Qwen3-8B and 100 to 30 on Llama-3.1-8B.

Breaking the stream while keeping the rehearsed positions fixed reverses the trade. Labels return — passage retrieval on Qwen3-8B rises from 59 to 78.7, level with KV2 at its default ratio on the same split — but needles leave, with RULER dropping from 81.8 to 69.8, multi-key-2 from 57 to 5, and multi-value from 98 to 59. Under a tight budget the two kinds of tokens compete for the same slots.

The advantage depends on what the questions ask for

The paper's own framing makes the boundary explicit: coherent rehearsal spends the budget on what the model cannot predict, while sparse rehearsal spreads it over everything rehearsed and crowds the needles out. So Attic's largest gains appear on fact-sparse tasks where questions target specific values, entities, and numbers, and its weakness appears where the answer is a predictable label that the model can produce from the stream itself.

The evidence covers RULER with four LLMs and LongBench's natural-text tasks with two, plus LooGLE for the compress-once-answer-many case, all in the 3–5% keep-ratio regime the paper targets. The reported comparisons are against training-free methods and against hosts trained as little as Attic; the abstract's claim that Attic alone is best in all eight settings is a training-free comparison, not a claim against every trained system.

The practical priority that follows is a choice about what to rehearse, not about how to score. If the cached context will be queried for specific facts, question-answer rehearsal with a content-adaptive amount is the configuration the paper's numbers favor, and its lead grows as the budget shrinks toward 3%. If the queries instead target predictable structural labels, the same coherence that keeps needles will lose those labels, and the paper's own sparse-rehearsal control shows the opposite trade.

Sources used in this answer