Can compact video encoders be built on frozen image representations?

Three operations that video encoders usually couple

The paper's starting point is that a video encoder decides two things at once: when frames begin to interact, and which frame-specific visual evidence stays reachable by the language model. Native video pathways, such as the Conv3D considered here, let neighboring frames contribute to shared features through spatiotemporal convolutions or attention before any question-aware selection happens. Image pathways instead keep each frame's representation independently computed, but forwarding all of those tokens costs far more visual tokens.

The authors argue these are not the same operation. Making a representation compact and keeping its decisive evidence accessible can pull in different directions. Their design therefore splits what is often bundled together into per-frame representation, cross-frame token allocation, and temporal interaction, and asks whether the third can be deferred until after the second.

Why early temporal mixing can hide decisive evidence

The motivating example is a clip where the same scene repeats but the question hinges on a brief change in a small region, such as the instant of take-off or contact at landing. Repeated background views cannot substitute for that instant, and a representation that spends its budget on repeated evidence loses the observation that distinguishes the states.

In the Conv3D pathway, neighboring frames contribute to shared features before question-aware selection, so the selector acts on temporally coupled tokens rather than separately addressable image tokens from each frame. The paper is careful about the claim: the concern is not that Conv3D inevitably erases motion, but that frame-specific evidence is no longer independently selectable. Independent image encoding exposes those candidates, yet forwarding them all leaves relevance and redundancy unresolved, since repeated backgrounds and persistent objects occupy token slots alongside the observations the question needs.

Selection sets the budget, refinement adds context

A frozen image encoder first produces frame-specific candidates. A question-aware selector then allocates a fixed token budget across frames using relevance, diversity, and cross-frame correspondence. The diversity term is a determinantal point process kernel built from normalized visual features and question relevance, while an adjacent-frame pair reward encourages keeping correspondence endpoints jointly. Frame quotas are inherited from a frozen upstream allocator rather than predicted anew, so the selector changes which source tokens occupy the slots, not the frame-level budget profile.

After selection, a lightweight learned refiner reads neighboring-frame context from the unchanged dense pool and writes residual updates only to the retained anchors. It adds roughly 18M trainable parameters, uses a bottleneck cross-frame attention block and a gated feed-forward network, and is trained for two epochs on 8 A100 GPUs for about 96 GPU-hours while the vision encoder, language model, and selector stay frozen. Because updates are residual and only the retained anchors are written to, source positions are preserved and the visual output stays at the fixed budget.

The cost of selecting before frames interact

The paper formalizes the trade-off with a diagnostic linear model rather than claiming the native backbone is linear. Comparing the two operation orders, interact-then-select versus select-then-interact, the difference is interaction across the selection boundary, not output length. The two orders agree for every input if and only if the cross-block interaction term is zero.

Reading context from the unselected pool restores part of that interaction, and the exact worst-case residual for the remaining unobserved indices scales with the norm of the omitted interaction block. The authors state this is not an accuracy bound for an adaptive neural encoder. Its design consequence is concrete: the method must avoid both spending output slots on repeated observations and making a selected observation unusable by cutting off its relevant context.

Matching full-image accuracy at 28%-35% of tokens

On Qwen3-VL-8B, the full pathway reaches a 13-benchmark macro-average of 62.75 with 1,535 visual tokens, compared with 62.58 for the full Image pathway at 4,424 tokens and 59.49 for native Conv3D at 2,212 tokens. On Qwen3-VL-32B, it reaches 66.28 against 66.09 for Image while providing a 2.16x end-to-end speedup. Across the three backbones the pathway matches full-image aggregate performance using about 28%-35% of the visual tokens.

The latency gain sits before autoregressive decoding. On Qwen3-VL-32B-Instruct, time to first token falls from 849.1ms for Image to 365.8ms, while post-first-token generation changes only from 62.1ms to 56.0ms; selection overhead is included in TTFT and is offset by the shorter visual prefill. End-to-end latency drops from 911.2ms to 421.8ms relative to Image.

Coordination and correspondence carry most of the gain

Ablations on Qwen3-VL-8B isolate the components. Removing dynamic allocation and cross-frame coordination gives a 13-benchmark average of 60.12; adding dynamic allocation alone raises it to 61.74; adding cross-frame coordination reaches 62.64. The learned refiner contributes a smaller complement: answer supervision alone reaches 62.67 and knowledge distillation alone 62.71, and combining both gives 62.75. The aggregate gain is modest, with Video-MME, LVBench, and Perception Test benefiting more strongly while VideoAds decreases slightly.

System-level ablations on Qwen2-VL-7B show that correct cross-frame correspondence matters: ignoring it drops the average from 59.43 to 58.50, and randomly shuffling correspondences drops it further to 57.53, suggesting incorrect pair structure can be actively harmful rather than merely uninformative. Broader candidate coverage helps monotonically over the evaluated range, from 57.97 at 16 frames to 58.85 at 24 frames and 59.43 at 32 frames.

Where the token budget stops paying off

The output budget controls the accuracy-compression trade-off. On Qwen2-VL-7B, cutting the budget from 1,536 to 768 tokens lowers the 13-benchmark average by 2.80 points, while raising it to 3,072 tokens adds 1.68 points. Across the three backbones, moving from 768 to roughly 1.5K tokens improves the average by 2.80, 2.60, and 2.13 points, but doubling again to roughly 3K yields smaller gains of 1.68, 1.50, and 1.57 points while increasing latency.

The advantage over prior compression baselines persists across a 4x range of output budgets: at 768 tokens the method exceeds the strongest prior baseline at the same budget by 2.83, 3.35, and 3.65 points, at 1.5K by 2.55, 2.89, and 2.42 points, and at 3K by 2.66, 2.75, and 2.75 points. The 1.5K operating point also outperforms every prior method evaluated at the 3K budget, at lower end-to-end latency. The authors treat approximately 1.5K tokens as the practical operating point because it matches prior compression baselines while retaining a substantial reduction relative to the full Image pathway.

Long-range subject binding remains the failure mode

The qualitative examples are meant to illustrate the kinds of evidence that matter under a compact budget, not to estimate how often a behavior occurs. Successful cases involve fine-grained local change, relational evidence that must be interpreted jointly, late or repeated-outcome evidence, and avoiding double-counting of repeated views.

The reported failure case stresses long-range subject binding rather than local appearance recognition: the question asks what the farther parrot does after spreading its wing near the end of the clip, and only the full Image pathway predicts the correct option while the compact pathways converge on the same distractor. The authors read this as a limitation of compact visual selection when the answer depends on maintaining instance identity across a long temporal span, and point to identity-aware retrieval and longer-range temporal state tracking as complementary directions. The training subset was also filtered against the evaluation prompts and metadata of all 13 benchmarks using exact n-gram matching and a semantic-similarity filter, which bounds how much of the reported advantage could come from train-test overlap.

Preserve evidence before compressing it

The paper's own summary of its design principle is that compact video encoding does not require early temporal mixing: frame-specific evidence can be preserved first, allocated jointly, and temporally contextualized after selection. The most decision-relevant comparison for a reader choosing an approach is therefore not raw token count but the pairing of a fixed output budget with coordinated, correspondence-aware selection.

Within the tested range, the practical threshold is the 1.5K-token operating point: it matches full-image aggregate accuracy, beats prior compression baselines at the same budget and even at 3K, and sits where returns from adding tokens begin to flatten. The scenario split that matters most is the one the failure case exposes: when an answer depends on tracking a specific instance across a long span among visually similar objects, compact selection on frozen image representations is the setting where the full Image pathway still retains evidence the compact pathway loses.

Sources used in this answer