How can token diversity and task relevance separately guide visual token pruning?

One pruning score mixes two different redundancies

Most visual token pruning methods rank tokens with a single score built from attention, feature similarity, or output divergence, then discard the lowest-ranked tokens. The paper argues that these scores silently blend two distinct sources of redundancy: intra-modal redundancy, where visual tokens duplicate each other, and cross-modal redundancy, where a visual token carries little information for the text query.

The distinction matters because the two cases are not comparable on one scale. A token can be visually unique yet irrelevant to the question, or highly relevant yet spatially duplicated. Any single scalar must trade one against the other implicitly, which the authors argue is often suboptimal. Their motivating example is attention-guided pruning that discards the white coffee-cup region when asked where the coffee cup is, producing a wrong answer.

The paper formalizes this by mapping visual and text tokens onto orthogonal subspaces and defining separate measures: a pairwise similarity kernel over visual tokens for intra-modal redundancy, and a relevance function between visual tokens and text for cross-modal redundancy. Under the orthogonality and decorrelation assumptions, the covariance between the two measures is zero, so they vary along independent statistical directions. The authors note this is a modeling assumption that holds approximately when the vision-language projector preserves modality heterogeneity, with the covariance bounded by the size of cross-subspace leakage.

Diversity selection keeps a representative visual subset

The first stage runs after the vision encoder and before the LLM. Instead of deleting redundant tokens, ToDRE selects a subset that is maximally diverse, using a greedy max-sum diversification procedure. It starts from a pivot token, then repeatedly adds the token whose cumulative cosine similarity to the already selected set is lowest, so each new token is as unlike the current set as possible. The process stops at k retained tokens, for example k=288, about 10% of the visual tokens.

The pivot is chosen by the highest [CLS] attention from the last vision-encoder layer, treating that attention as an importance indicator. An ablation on LLaVA-NeXT-7B shows this choice works best, but random pivot selection performs nearly comparably, which the authors present as a practical fallback for models whose encoders lack a [CLS] token. At a 10% retention ratio, the [CLS] pivot reaches 96.0% average performance, while random, center, and farthest pivots reach 95.8%, 95.6%, and 95.8% respectively.

This stage addresses intra-modal redundancy only. The retained tokens are diverse and representative of the image, but nothing yet guarantees they are the ones the question needs. That is left to the second stage.

Cross-modal attention fades before the final layers

The second stage relies on an observed pattern the paper calls information migration. Measuring text-to-visual and visual-to-text attention across LLM decoder layers, the authors report that cross-modal attention is high early and then declines, so visual information is largely absorbed into text representations in the deeper layers. During decoding, attention from output tokens to visual tokens stays below 5% in middle and deep layers, while attention to system prompt and text tokens remains high.

ToDRE uses this pattern as a removal signal rather than a per-token score. At selected deeper layers it computes two global cross-modal attention ratios, text-to-visual and visual-to-text, averaged over all heads and tokens. When both fall below a threshold tau, the layer is treated as the point where cross-modal interaction has sufficiently diminished, and all remaining visual tokens are removed from that layer onward.

Layer choice is a real trade-off. Applying the reduction from layer L/2 drops average performance to 87.9% of the full model, because early cross-modal attention ratios fluctuate and pruning interrupts ongoing alignment. Starting at 5L/8 or 6L/8 saves more time but costs roughly 2-4% average performance. Starting at 7L/8 restores performance to the baseline level while keeping nearly the same efficiency, which is why it is the default.

The threshold sets how aggressively tokens disappear

The threshold tau controls how extreme the second-stage pruning is. A larger tau triggers removal earlier and more aggressively, while a smaller tau makes pruning more conservative and preserves accuracy at higher computational cost. The ablation on LLaVA-NeXT-7B shows tau=0.03 at 99.9% average performance and 79:22 total time, tau=0.05 at 100.0% and 73:24, tau=0.10 at 100.1% and 72:35, and tau=0.15 at 100.0% and 72:25, against 77:04 for the uncompressed baseline.

The default tau=0.10 is chosen as the best trade-off: it keeps average accuracy at 100.1% of the baseline while cutting more than four minutes of inference time. These numbers come from a four-benchmark average (MME, ScienceQA, GQA, POPE) on one model, so they describe this configuration rather than a universal operating point.

Reported gains depend on which stage is counted

The headline result combines both stages: ToDRE prunes 90% of visual tokens after the vision encoder and all remaining visual tokens in certain LLM decoder layers, yielding a 2.6x speed-up in total inference time while maintaining 95.0% model performance. The discussion section reports experiments on twelve image- and video-language benchmarks, and adds 14.5% lower memory usage than uncompressed baselines.

These figures should not be read as one uniform measurement. The 90% figure refers to tokens removed after the vision encoder, the decoder-layer removal is a separate all-token deletion at a selected layer, and the 95.0% performance figure is an aggregate across benchmarks rather than a per-task guarantee. The paper also reports that ToDRE consistently outperforms all baselines across eight image-language benchmarks, which is a comparison claim, not a claim that every individual benchmark improves.

The method is training-free and architecture-agnostic, so it does not require retraining the LVLM. That property is what makes the pruning cost mostly inference-time bookkeeping, but it also means the method inherits whatever redundancy structure the pretrained model already has.

Where the two-stage design is most worth trying

The paper's own ablations point to a specific priority: spend the first stage on diversity, because that is where most tokens go, and use the second stage only in the deeper decoder layers where cross-modal attention has already faded. Pruning earlier in the decoder is the setting that most clearly damages performance, so the layer threshold, not the retention ratio alone, is the variable to watch when adapting ToDRE.

Two practical conditions follow from the evidence. Models without a [CLS] token in their encoder can fall back to random pivot selection with only a small measured drop, which broadens where the first stage applies. And because the second stage depends on measured attention ratios falling below tau, the appropriate threshold is tied to the model and benchmark mix used for calibration; the reported 100.1% average at tau=0.10 is evidence for that configuration, not a value to copy blindly.

The most useful way to read this paper is as a separation argument: diversity and task relevance answer different questions, and the evidence shows they can be handled in different places in the pipeline without retraining. The decoder-layer threshold is the knob that decides whether that separation buys speed or costs accuracy.

Sources used in this answer