How does query-adaptive memory refinement improve multi-agent long-video understanding?

Two memory paradigms fail in opposite directions

Answering questions about hour-long videos requires holding global context while still locating decisive evidence at the right semantic granularity. The paper frames existing methods as two paradigms with complementary weaknesses. Top-down, query-driven exploration starts from the question and inspects relevant intervals, building or updating hierarchical memory during search; it adapts observation to the question, but inaccurate localization can send the search away from answer-critical segments. Bottom-up, query-independent memory construction builds a memory first and retrieves or revisits at inference time; it gives broad coverage but may omit details a specific question needs.

The paper's own illustration makes the split concrete. In a Tom-and-Jerry chase example, the top-down path answers "Not Previously Stored" after a localization error, while the bottom-up path answers "On a wooden crate" instead of the ground-truth "Beside his mouse hole." VideoTapestry's premise is that these paradigms are complementary: query-independent memory supplies coverage, and query-driven exploration supplies question-specific evidence, provided the two can be merged without losing global context.

Three memory levels map to three specialized agents

VideoTapestry instantiates a fixed, query-independent hierarchy following the memory-construction procedure of MemDreamer, then refines it at query time. The hierarchy has three semantic levels: SuperEvents summarize coarse temporal stages, MacroEvents describe localized coherent events, and each MacroEvent carries a local Subgraph of Entity, Event, and OCRText nodes connected by typed relations. The paper assigns these levels distinct roles: global narrative context, event-level temporal structure, and fine-grained factual evidence.

Each level gets its own agent so that retrieval and refinement stay inside a scale-specific context. The Overview Agent refines the Super-level scaffold and selects relevant stages; the Skim Agent examines their MacroEvents and narrows candidate regions; the Focus Agent inspects the corresponding Subgraphs for fine-grained verification. Selected parent identifiers determine which descendants are exposed at the next level, so the agents form a coarse-to-fine chain rather than three independent retrievers.

Crucially, refinement modifies only temporary textual representations. Node identifiers, parent relations, semantic levels, and temporal ranges stay fixed, and the underlying static hierarchy is preserved. That is what allows the refined layers to be reassembled under their original parents later.

Each agent plans, observes, updates, then decides

All agents follow the same Plan, Observe, Update, Decide protocol. Plan identifies evidence needs, Observe inspects selected intervals, Update integrates new evidence into temporary memory, and Decide selects the next scope or terminates. Across all benchmarks, each visited layer performs one refinement batch followed by one layer-level decision.

Observation is tool-mediated. Every level can call a Visual Observation Tool that maps a set of temporal intervals and an inspection instruction to visual observations. The Overview Agent additionally has a Transcript/ASR Tool that extracts spoken evidence from timestamped text in selected windows; in the evaluated Video-MME configuration it uses Whisper transcripts and runs after visual observation on each successful Overview pass, while the Skim and Focus Agents use visual observations only.

The Overview Agent's update is asymmetric. Evidence-associated SuperEvents and their immediate temporal neighbors form a protected set that keeps its original descriptions and receives any associated visual, transcript, and verbatim evidence; all remaining SuperEvents are compressed into compact summaries. The protected set is enforced programmatically, so protected neighbors without observations simply retain their original text. This is the mechanism that keeps video-wide coverage while concentrating detail where the query points.

Assembly keeps the scaffold and enriches only relevant branches

After traversal, the Assemble step retains the refined Super-level scaffold and places the selected refined MacroEvent and Subgraph memories beneath their original parents. The result is a composite query-adaptive memory that preserves video-wide structure while enriching query-relevant branches. The paper describes this as preserving global context in compact form while retaining fine-grained evidence along the branches the query actually needs.

The memory is then serialized together with the question, the options, and a compact handoff trace that records each agent and its handoff instruction or stopping marker. The trace may summarize intermediate findings and unresolved evidence needs, but it excludes complete observation and update logs. An independent Answer Agent predicts from this serialized context without further retrieval or observation, which separates the refinement stage from the final reasoning stage.

Termination is governed by a validated stop flag. In the evaluated Video-MME configuration, global questions may terminate at an intermediate level, whereas non-global questions must descend when valid descendants are available, and the Focus Agent always terminates exploration.

Gains over direct GPT-5.5 range from 7.0 to 17.2 points

The headline comparison is against direct GPT-5.5 inference. VideoTapestry reports absolute accuracy gains of 17.2 percentage points on LVBench, 14.9 on LongVideoBench (Long), 9.8 on Video-MME (Long), and 7.0 on EgoSchema, and the paper states it achieves state-of-the-art results among all competitors on these four long-video understanding benchmarks. These are accuracy values in percent, and the reported figures are percentage-point differences over the direct baseline, not relative improvements.

The comparison is deliberately controlled. Direct baselines receive uniformly sampled individual frames without grid packing, and the paper notes that grid packing is part of VideoTapestry's coarse observation mechanism rather than generic preprocessing, so applying it to the direct baseline would add a temporal-compression strategy beyond standard frame prompting. Under the API gateways' maximum image-upload limits, GPT-family models receive 50 frames, Qwen3.7-Plus receives 196 frames, and Qwen3.6-27B receives 396 frames per video. GPT-family models run at high reasoning effort and Qwen-family models in thinking mode for direct inference, with temperature 0.1 throughout.

The paper also reports that VideoTapestry consistently improves over the same-backbone direct baseline and over representative agentic methods. The same-backbone end-to-end evaluation uses frozen GPT-5.5, and some baseline values come from models' technical reports or previously published benchmark results rather than from the authors' own runs.

Two trajectories show refinement correcting coarse errors

The first qualitative case is fine-grained temporal verification. Asked what a man does from 00:27 to 00:33, the Overview Agent inspects the backstage stairwell and door sequence, finds a plausible door event but no direct match for climbing direction, and notes that ASR does not establish movement direction. The Skim Agent narrows to a macro event and confirms downward action and a closed door. The Focus Agent then verifies state by state: upper stair at 27.5 s, descending at 29.5 s, reaching the landing at 30.5 s, and a closed door at 32.5 s, concluding that no door or window opening occurs. The final verdict is C, walking down the stairs.

The second case is cross-branch evidence aggregation. Asked how many contestants are seen playing the piano, the Overview Agent finds two plausible piano events in distant temporal regions, roughly 25 minutes apart, but cannot resolve whether the appearances repeat. The Skim Agent inspects both branches and finds visually distinct contestants at different pianos. The Focus Agent verifies local keyboard-playing evidence in each branch and confirms they are distinct contestants rather than repeated shots, yielding a total of two and the answer D.

These trajectories illustrate the mechanism the paper claims: coarse localization proposes candidate regions, and finer levels either confirm or correct them. They are qualitative examples rather than a separate quantitative evaluation, so they support the described behavior rather than adding new accuracy numbers.

Training-free design shifts cost to inference-time tooling

VideoTapestry is training-free, but it is not free of infrastructure. Each Visual Observation Tool call uploads at most 50 images to respect the GPT API gateway limit; the Overview Agent samples up to 120 frames packed into at most 20 six-frame grids, while Skim and Focus each use up to 50 individual frames from selected intervals. The ASR Tool applies Whisper only to agent-selected audio windows rather than transcribing the full video, accessing at most three windows, each normalized to 600 to 1200 seconds within video bounds. For benchmarks with provided subtitles, the same temporal-selection procedure retrieves subtitle text instead of invoking Whisper.

The paper also isolates its query-time method from offline memory construction: all experiments use hierarchical video memories built following the MemDreamer procedure, and the MemDreamer baseline's retrieval pipeline is reproduced separately. That means the reported gains measure query-adaptive refinement on top of a fixed preconstructed hierarchy, not the hierarchy itself. For GPT-family models only the Answer Agent uses high reasoning effort, and for Qwen-based experiments thinking is enabled only for planning, decision, and final answering.

Because the evidence packet contains the abstract, introduction, methods, results, and discussion rather than the full appendix, the detailed stopping rules, decision validation, and complete per-benchmark tables referenced in the text are not available here. The accuracy gains and the mechanism are documented; the finer configuration choices behind them are only partially visible.

Refinement pays off when the query needs specific evidence

The paper's own framing sets the decision threshold. If a question can be answered from broad, query-independent coverage, a bottom-up memory already suffices, and the extra refinement passes add little. The refinement machinery is designed for questions whose decisive evidence is narrow and localized, where top-down exploration risks a localization error and static retrieval risks omitting the needed detail. The protected-set update is the concrete expression of that priority: it compresses everything outside the evidence-associated SuperEvents and their immediate neighbors, spending detail only where the query points.

The benchmark pattern is consistent with that reading. The largest gain, 17.2 points on LVBench, and the smallest, 7.0 on EgoSchema, are both measured against the same direct GPT-5.5 baseline, so the spread reflects how much each benchmark's questions benefit from query-adaptive refinement rather than a difference in method. The two qualitative cases show the same logic at work: a six-second interval that requires state-by-state verification, and two piano branches roughly 25 minutes apart that require aggregating evidence across distant regions.

The practical takeaway is therefore conditional rather than universal. The value of VideoTapestry comes from combining a preserved global scaffold with selectively enriched query-relevant branches, and the reported gains apply to long-video question answering evaluated on these four benchmarks with a frozen GPT-5.5 backbone and a MemDreamer-built memory. Whether the same refinement helps shorter videos, different memory-construction procedures, or questions that are already well served by uniform sampling is not something this evidence packet establishes.

Sources used in this answer