[CVPR 2026] ColParse: Breaking the Multi-Vector Storage Bottleneck with Layout-Informed Retrieval

TL;DR

Multi-vector retrieval (pioneered by models like ColPali) is the gold standard for Visual Document Retrieval (VDR) but is notoriously "storage-hungry." ColParse changes the game by replacing uniform grids with layout-aware sub-images. By fusing local semantic regions (tables, text blocks) with global context, it achieves 95%+ storage compression and SOTA performance without any additional training.

Problem: The Cost of Fine-Grained Vision

Visual documents (PDFs, reports, invoices) are messy. Traditional "Single-Vector" models compress an entire page into one embedding, often losing the "needle in the haystack" (like a specific number in a table). "Multi-Vector" models solve this by storing embeddings for every small patch of the image.

The catch? Storing 700-1000 vectors per page makes large-scale deployment impossible for most enterprises. Previous attempts to fix this—like clustering patches or pruning vectors—often "blind" the model to the document's physical structure.

Figure 1: Comparison of multi-vector VDR optimization strategies.

Methodology: Beyond the Grid

ColParse's core insight is that layout is the document's natural index. Instead of a $32imes32$ grid of arbitrary patches, ColParse identifies the actual components of the page.

1. Layout-Informed Parsing

Using a high-performance parser (MinerU2.5), the system identifies $<10$ key regions (e.g., Title, Table 1, Paragraph 3).

2. Dual-Stream Encoding

• Local Stream: Each sub-image is encoded into a specific vector.
• Global Stream: The entire page is encoded to capture the "big picture."

3. Global-Local Fusion

A critical step. A local table vector might lose context (e.g., which year it refers to). ColParse fuses the global vector into each local vector: $d_{fused}=\alpha \cdot v_{global}+(1-\alpha )\cdot v_{local}$ This ensures every local vector is "context-grounded."

Figure 2: The ColParse framework workflow.

Experiments: More Performance, Less Space

The authors tested ColParse across 24 datasets and 10 different base models (VLM2Vec, GME, UniME, etc.).

• Performance: On the ViDoRe-V1 benchmark, ColParse improved VLM2Vec-V1-7B by an incredible 42.69 points in nDCG@5.
• Efficiency: Compared to the multi-vector SOTA (ColQwen), ColParse achieves higher accuracy while reducing the vector count from 768 down to roughly 6 per page.

Figure 3: Performance comparison showing ColParse (red envelope) consistently outperforming baselines.

Deep Insight: Why Fusion Matters

The "Ablation Study" highlights that simply keeping sub-images isn't enough. Without the global vector fusion, the model struggles with queries that require page-wide understanding. The fusion acts as a "semantic anchor," allowing the late-interaction mechanism (MaxSim) to find regional details without losing the document's overall theme.

Conclusion & Implementation

ColParse is a training-free, plug-and-play module. It proves that for visual documents, we don't need more vectors—we need smarter ones.

Key Takeaways for Practitioners:

1. Parser Quality is King: Using MinerU2.5 was central to their success due to its industrial-grade speed (2.25 Pages/sec).
2. Interpretability: Because vectors are tied to layout blocks, the system can tell the user exactly which part of the page (e.g., the specific table) triggered the retrieval.
3. Scalability: It makes Multi-Vector retrieval viable for million-page corpora.