Can Streaming VLMs Perceive High-Dynamic Real-World Video Streams?

Trajectories and humans filter weak questions

The construction pipeline starts from high-FPS clips and generates candidate question-answer pairs. A candidate is discarded if the question can be answered from a 2 FPS subsample, which is the step that keeps the benchmark focused on high-dynamic perception rather than ordinary video comprehension. Retained answers are then checked against object trajectories from SAM3 segmentation masks and CoTracker3 tracking, and pairs that cannot be verified or corrected are dropped. Three rounds of human inspection follow, and human experts made the final accept-or-reject decision for every released sample.

This matters for reading the scores: a wrong answer on FastBench is meant to indicate a genuine perceptual miss on a verified fast event, not an ambiguous or poorly annotated question. The paper also discloses that Gemini-3.1-Pro was used to propose keywords, draft candidate pairs, and run the trajectory checks, and that Gemini-3.5-Flash read timestamped comments and danmaku when localizing candidate segments. The authors state they reviewed model-generated annotations against the source videos and trajectory evidence.

Why more frames can mean less history

Streaming VLMs process arriving video continuously under a bounded context budget. The paper fixes a unified maximum of 248,832,000 pixels, equivalent to 120 frames at 1080p. Within that ceiling, a model must divide capacity among three competing needs: how far back in time it can see, how much spatial detail it keeps, and how finely it samples in time. Spending more on frame rate necessarily shortens the historical window or lowers resolution.

This is the mechanism behind the paper's central empirical pattern. Dense sampling is not free: it buys fine temporal detail at the cost of the memory a model needs for backward-looking questions. The authors test this directly by varying input frame rate for Qwen3-VL-8B while holding the context budget fixed.

Denser sampling peaks at 24 FPS, not 30

For Qwen3-VL-8B, overall accuracy rises monotonically from 32.9% at 2 FPS to 44.6% at 24 FPS, a gain of 11.7 percentage points. Native 30 FPS then falls back to 42.1%. The authors attribute the drop to the trade-off above: maximal frame rate severely compresses the historical receptive field and adds substantial inter-frame redundancy, so 24 FPS offers the better balance between context depth and high-frequency detail.

The breakdown by capability and scope shows where the gain and the loss land. Motion and spatiotemporal tracking improves by 13.7 points and predictive and causal reasoning by 14.5 points, both peaking at 24 FPS. The 30 FPS drop is most pronounced for backward queries, which fall from 46.2% at 24 FPS to 37.5%, consistent with their reliance on historical evidence. Instant queries, which depend mainly on immediate frame resolution, remain strong at 30 FPS at 46.3%. These are results for one open-source model under one fixed budget, not a universal optimum.

Knowing when to look closer is the bottleneck

The paper also introduces ProactiveFrame, a training-free baseline that lets the model control its own sampling. The model emits text tokens: Focus_Start requests a higher frame rate for the next chunk, Focus_End returns to normal sampling, and Silent continues the current mode without answering. A dual-tier sliding window keeps recent high-FPS observations while downsampling older ones into sparse history. The focus window is set to 90 frames at 1080p, leaving capacity for at least 30 low-FPS frames of sparse history.

ProactiveFrame beats the base models, raising Qwen3-VL-8B from 32.9% to 38.3% and Gemini-3.5-Flash from 50.7% to 52.2%. It also outperforms sparse uniform sampling by 5.4 and 1.5 percentage points respectively. But it remains below an oracle that is handed the human-annotated focus intervals: 40.7% for Qwen3-VL-8B and 57.5% for Gemini-3.5-Flash. The gap isolates the failure: models can use a higher frame rate when given the right moment, but struggle to decide from the stream alone when finer temporal perception is needed.

For Qwen3-VL-8B, full 30 FPS beats the oracle

One comparison complicates a simple "sample denser" reading. For Qwen3-VL-8B, processing the stream at its native 30 FPS throughout reaches 42.1%, higher than the oracle mixed-FPS setup at 40.7%, which supplies 30 FPS only inside annotated focus intervals and 2 FPS elsewhere. The authors read this as a bottleneck in current open-source VLMs: even with perfectly isolated high-FPS segments, their fine-grained dynamic understanding remains limited, so the visual context lost by downsampling background frames outweighs the benefit of longer history.

The pattern differs by model and by scope. Gemini-3.5-Flash peaks under full 30 FPS at 55.3%, above both its ProactiveFrame score of 52.2% and the oracle's 57.5%? No: the oracle is higher for Gemini. For forward queries, full 30 FPS exceeds mixed FPS for both models (56.3% versus 55.4% for Gemini-3.5-Flash and 42.0% versus 39.4% for Qwen3-VL-8B). Backward queries peak under the oracle for both models (55.4% and 40.6%), matching the idea that retrieving past evidence is most sensitive to the memory-resolution balance the oracle strikes best.

What the 50.7% ceiling does and does not show

The headline number is a percentage of correctly answered FastBench questions under the paper's unified context budget and its LLM-as-a-Judge scoring of open-ended answers, with a relaxed late-response boundary that extends the valid window to 4 seconds after an event concludes. It is not a claim about all streaming video understanding, and the benchmark deliberately excludes questions answerable at 2 FPS, so it measures the high-dynamic slice rather than average performance. The frame-rate ablation covers Qwen3-VL-8B; the context-formulation comparison covers Qwen3-VL-8B and Gemini-3.5-Flash.

The practical priority the paper establishes is therefore not simply raising FPS. Under a fixed budget, 24 FPS was the best point for the one model swept across frame rates, and the oracle gap shows that the larger remaining gain lies in proactive temporal control: deciding when a scene warrants dense observation. The authors position FastBench and its annotated focus timestamps as a testbed for that problem, and the comparison that matters most is ProactiveFrame against the oracle, not against 2 FPS.

Sources used in this answer