When Does Visual Token Pruning Improve Calibration in MLLMs?

The answer depends on which tokens the selector keeps

Pruning methods are usually compared by the accuracy they retain at a fixed token budget. This paper asks a different question: after pruning, does the model's confidence still tell you when its answer is likely wrong? Across nine selectors and five models from the LLaVA-1.5 (7B, 13B) and Qwen2.5-VL (3B, 7B, 32B) families, the answer splits by selector family rather than by model size.

The mechanism the authors propose is simple. The language model was trained on complete token sets, and a pruned input carries no signal that tokens are missing. If the removed tokens held the evidence for the answer, the model answers from what remains, and its confidence reflects only those remaining tokens. The paper tests this by recording which tokens each selector keeps and, on POPE, measuring how much of the queried object survives.

When the queried object loses all its tokens

The clearest evidence comes from questions where the object disappears completely. With seven selectors at 64 or 128 tokens, the queried object lost all of its tokens in 4,557 answers from LLaVA-1.5-7B. On those same questions, the pruned model answered 17% correctly, against 59% for the unpruned model. Its mean confidence on those questions was 0.82, essentially the unpruned level of 0.80.

The paper's illustrative case is a POPE question about a bus that fills much of the image. A selector keeping the 64 most attended tokens retained 3% of the bus area, and LLaVA-1.5 answered that there was no bus with confidence 0.96. A selector spreading 64 tokens over the image retained 12% of the bus, and the model answered correctly.

Confidence here is not a separate accuracy measure. It is the model's stated probability for its answer, and the point is that it fails to register the removal of evidence: the accuracy drop is large while the confidence drop is negligible.

Restoring a few object tokens recovers most accuracy

To test whether the removed evidence actually causes the errors, the authors edit the kept set rather than the model. For CLS attention and FastV at 64 tokens, they take the 504 and 1,260 questions whose object lost all of its tokens and replace up to eight randomly chosen kept tokens with the grid cells holding the most object area. A control replaces the same kept tokens with random cells that are neither kept nor in the object.

Relative to that control, accuracy changes by +48.4 points (95% interval 41.5 to 55.9, image-clustered bootstrap) for CLS attention and +59.4 (54.8 to 64.1) for FastV, while mean confidence moves by only −0.04 and −0.07. Restoring at most eight cells brings back 91% and 87% of the object area on average. Returning the same number of other tokens does not help.

The reverse intervention runs on coverage selection, using 3,567 questions whose object kept between 1 and 32 cells. Removing those kept object cells drops accuracy by 50.2 points (−53.0 to −47.3) relative to a control that removes the same number of kept cells outside the object, with confidence falling only 0.09. The budget stays at 64 tokens in every condition, so the comparison isolates which tokens are kept, not how many.

Attention-ranked selectors produce confident errors

Selectors that keep the most attended tokens remove the queried object more often and barely lower confidence on the resulting errors. On POPE with LLaVA-1.5-7B at 64 tokens, FastV reaches 71.1% accuracy at a mean confidence of 0.92, while the unpruned model reaches 87.0% at 0.91; CLS attention sits at 80.5% accuracy and 0.89 confidence. Coverage selection and CDPruner land near the diagonal of the reliability diagram, at 85.1% and 88.0% accuracy respectively.

The same pattern appears in the full grids. On Qwen2.5-VL-7B POPE at 11% of tokens, FastV has an expected calibration error of 0.185 against 0.082 unpruned, and on GQA with whole-answer probability it reaches 0.286 against 0.200. Coverage and DivPrune raise the unpruned ECE by at most 0.024 across LLaVA-1.5-13B and Qwen2.5-VL at 22% of tokens.

Temperature scaling, the standard post-hoc fix, brings every ECE to 0.07 or less. But it is a monotone map, so it cannot change how confidence ranks right and wrong answers. For FastV with 64 tokens, the AUROC of confidence for correctness falls from 0.801 to 0.674, and that ranking damage survives rescaling.

Why coverage selection stays near the unpruned model

Coverage- and diversity-based selectors avoid keeping redundant tokens, spreading the budget across the image instead of concentrating it on the most attended regions. That is why part of the queried object usually survives: in the intervention, coverage selection retained 20% of the object area on average, while CLS attention and FastV retained 0%.

The calibration consequence follows from the same mechanism. On POPE, coverage selection with 128 tokens matches unpruned accuracy within one point and lowers ECE from 0.040 to 0.017. In long image descriptions, the removed evidence shows up as objects the pruned model simply does not mention, which is the observable form of the same failure outside yes/no questions.

The paper's evidence measurements rely on COCO masks, so for open-ended answers the pattern is observed only through calibration and omitted objects rather than through direct token-level accounting. That limits how precisely the mechanism can be traced outside POPE-style questions.

Confidence can set a per-question token budget

The second half of the paper turns the diagnosis into a serving scheme. Each question is first answered with a small budget, here 64 tokens, and escalates to the unpruned model only when confidence falls below a threshold. Conformal risk control picks that threshold so the expected share of answers differing from the unpruned model stays below a target ε, treating images as the exchangeable units because benchmarks ask many questions per image.

The bound holds for every selector, but the cost of meeting it does not. On POPE with ε = 2% and a first budget of 64 tokens, coverage selection needs 36% of the unpruned model's prefill tokens and answers 2.3 times faster in batches of 32. FastV needs 105%, more than running the unpruned model directly, because its confident errors force escalation.

The saving is not automatic from pruning. With an uninformative score, coverage selection would need 97% of the prefill tokens; the gap between 97% and 36% is what its confidence ranking buys. Across five models, the confidence of coverage- and diversity-based selectors realizes 71% to 95% of the saving between an uninformative and a perfect gate on POPE, but only 31% to 54% on GQA; FastV realizes 18% to 43% and 18% to 25%.

Where the cascade pays off and where it does not

The cost figures are prefill tokens, the input tokens the language model processes before answering, and a question that escalates costs more than running the unpruned model directly because it pays for both stages. That accounting explains why FastV loses: its first stage is cheap, but its confident errors send too many questions to the second stage.

Design choices shift the balance by selector. Starting the cascade at 22% instead of 11% of tokens lowers cost for attention-ranked selectors and random selection on POPE, for example from 0.73 to 0.55 for CLS attention, but raises it for coverage and diversity selectors, whose 11% first stage already answers more than 80% of POPE questions. Passing through three budgets before the unpruned model costs more than the two-stage cascade for every selector and benchmark.

Latency is a separate ledger. At batch size one on an H200, the unpruned model takes 52.7 ms per POPE question, while FastV, VisionZip, and PDrop save 4 to 7 ms. The input-level selectors run the vision encoder a second time to obtain selection features, and that pass plus greedy selection outweighs the saving: coverage selection with 128 tokens takes 72.2 ms. Throughput gains appear at larger batches, where 64 visual tokens give 5.0 times the throughput of 576 tokens at 32 or more questions per batch.

Report calibration next to accuracy and cost

The paper's practical claim is a comparison rule rather than a universal speedup. Two selectors can retain similar accuracy at the same budget and still differ sharply in whether their confidence is usable: on POPE at 64 tokens, FastV and coverage selection sit at 71.1% and 85.1% accuracy, but only coverage selection keeps confidence informative enough to gate escalation cheaply.

The validity checks bound how far the risk guarantee travels. On ScienceQA-IMG, where each question has its own image, the average test deviation over 200 splits lies within one standard error of ε = 2%, exceeding it by at most 0.0006 across designs and targets. On POPE and GQA, with 250 and 199 calibration images, the correction term is 0.004 to 0.005 and the average deviation stays below ε in every configuration. The bound assumes exchangeable images, so it is a statement about expected deviation averaged over an image's questions, not a per-question guarantee.

The decision this paper supports is therefore conditional: if a pruned model's confidence will drive abstention, routing, or a token budget, the selector's evidence coverage matters as much as its retained accuracy, and calibration belongs in the comparison table next to accuracy and cost.

Sources used in this answer