Can dynamic vector decoding make MLLM perception more token-efficient?

Coordinate text buys precision with tokens

Multimodal large language models (MLLMs) handle perception by emitting the coordinates of a box or mask as text. That representation has no minimum quantization unit, so it can express any range at any precision, but every digit and separator is a token the model must generate and the language model must learn to parse. The paper frames this as a double cost: heavy token overhead at inference, and a harder learning problem because the model has to recognize the structure of floating-point numbers.

The common alternative is fixed-range quantization, which normalizes coordinates into a fixed integer range such as 0 to 1000 and maps them to special tokens. This shortens the output, but the paper argues it inherits two constraints that matter more as tasks move from 2D to 3D: a bounded numerical range and a floor on precision set by the smallest quantization step. In 2D the finite extent of an image partly hides these limits; in 3D, where spatial range is treated as unbounded and localization must be precise, the paper says fixed-range schemes either miss part of the scene or lose accuracy.

One vector sequence for 2D and 3D outputs

DVD's move is to stop treating coordinates as text and instead treat a perception output as a vector. The paper converts heterogeneous outputs, specifically 2D bounding boxes, 2D masks and 3D bounding boxes, into a 1D vector sequence, then maps that sequence to compact discrete tokens in a high-dimensional space. A lightweight de-tokenizer reverses the step, decoding the model's output tokens back into the original 2D or 3D representation.

The design borrows its premise from vector-quantized autoencoders, which learn compact discrete codes for continuous data through a limited codebook. The paper's stated argument is that a compact representation of a low-dimensional space can be achieved in a high-dimensional space. In practice this means the MLLM generates a handful of learned codebook indices rather than a coordinate string, and the de-tokenizer turns those indices back into geometry. The same pipeline is applied to boxes and masks and to both 2D and 3D, which is what the paper means by a unified representation.

Token counts, latency and accuracy move together

The most decision-relevant comparison is the paper's per-object measurement of tokens, latency and performance under identical training settings and compute. On 3D grounding, text coordinates reach 37.5 AP3D using 74 tokens and 4165 ms per object; special tokens drop to 17.5 AP3D with 15 tokens and 713 ms; DVD reaches 34.1 AP3D with 8 tokens and 499 ms. On 2D grounding, text scores 84.5 P@50 with 15 tokens and 750 ms, special tokens 83.2 with 4 tokens and 274 ms, and DVD 87.1 with 2 tokens and 143 ms. On 2D referring expression segmentation, text scores 48.0 cIoU with 512 tokens and 24410 ms, special tokens 42.9 with 128 tokens and 8771 ms, and DVD 56.3 with 64 tokens and 1715 ms.

These are three different metrics on three different task settings, so they should not be collapsed into a single efficiency number. What they jointly show is where DVD sits: it does not beat text on 3D grounding accuracy, but it comes within 3.4 AP3D while using roughly one-ninth the tokens and about one-eighth the latency, and it beats both baselines on 2D grounding and 2D segmentation. The paper's own reading is that text retains a slight 3D accuracy advantage because it preserves raw coordinate values with unlimited range and precision, while special tokens lose heavily to range and precision limits.

Benchmark results across 2D and 3D tasks

On the RefCOCO series, DVD-8B reaches 75.4, 76.9 and 74.6 on RefCOCO val, testA and testB; 70.1, 74.7 and 65.9 on RefCOCO+; and 73.4 and 76.6 cIoU on RefCOCOg val and test. The paper compares the RefCOCOg test figure with VistaLLM-7B and attributes the lower latency to reduced token overhead. The smaller DVD-2B scores 69.9, 70.7 and 69.7 on RefCOCO and 70.4 and 71.0 on RefCOCOg, so the gains are not uniform across model sizes.

The 3D evaluation spans four benchmarks chosen to cover indoor and outdoor settings: SUN-RGBD and Hypersim for indoor scenes, KITTI and nuScenes for outdoor driving scenes. The paper also reports a reconstruction check that isolates the autoencoder from the language model: decoding quantized tokens back to geometry yields 90.8 AP3D for 3D grounding, 100.0 precision for 2D grounding and 84.8 cIoU for 2D referring segmentation. That check matters because it shows the token bottleneck itself preserves spatial information before any MLLM reasoning is involved.

Codebook size sets the compression limit

The ablations explain why the default configuration looks the way it does, and they are run on DVD-2B with 30% of full training iterations, so they are diagnostic rather than final numbers. Codebook size is the clearest trade-off: 2048 entries underfits diverse geometry (88.9 AP3D, 84.5 precision), 8192 entries makes quantization and codebook learning harder and degrades reconstruction (85.1 AP3D, 76.8 cIoU), and 4096 entries is best across all three tasks and becomes the default.

Downsampling shows a similar cliff. Compared with no compression, a scale of 2 halves the token count while holding 90.8 AP3D and 100.0 precision and even slightly improving 3D grounding; a scale of 4 causes severe information loss, with 2D grounding precision falling to 31.5. Latent size behaves the same way: 8 is too narrow (83.0 AP3D), 32 brings no consistent gain and slightly lowers 3D quality (88.4 AP3D), and 16 is the stable middle. The geometric loss is the one component the paper calls essential rather than merely helpful, adding 10.1 AP3D on 3D grounding and 10.0 cIoU on 2D segmentation, with only 0.9 precision on 2D grounding where performance is already near saturation.

Where the unified design holds

Joint training is the test of whether one representation can serve both domains. Training the autoencoder on a single task gives 84.0 AP3D for 3D grounding alone, 100.0 precision for 2D grounding alone and 80.3 cIoU for 2D segmentation alone; training on all three together gives 90.8, 100.0 and 84.8. The paper reads this as joint training improving 3D grounding and 2D segmentation without degrading 2D grounding, which supports the claim that the shared vector representation does not force a trade between tasks.

The paper's own limitations bound this. Reconstruction quality could still improve at larger downsampling factors, and the de-tokenizer is trained as a separate component from the MLLM, which the authors say may leave the learned discrete tokens suboptimally aligned with the model's semantic understanding. Both caveats sit on the method itself rather than on the benchmark protocol, so they qualify how far the token savings can be pushed before the representation, not just the evaluation, becomes the bottleneck.

When DVD is the better choice

The paper's evidence supports a scenario split rather than a blanket verdict. Where output length and latency dominate, as in 2D referring segmentation with 512 text tokens per object, DVD's 64 tokens and 56.3 cIoU against text's 48.0 make it the stronger option on both axes. Where 3D localization precision is the binding constraint, text still wins on accuracy, and DVD's case rests on accepting a 3.4 AP3D gap in exchange for roughly an order-of-magnitude reduction in tokens and latency.

The configuration choices reinforce that this is a tuning problem, not a fixed property: 4096 codebook entries, downsampling scale 2 and latent size 16 are the paper's best trade-offs, and pushing compression further degrades reconstruction sharply. The practical reading is that DVD is most compelling when a system can tolerate a small 3D accuracy concession for large token and latency savings, and least compelling when full-range, high-precision 3D coordinates are the priority.

Sources used in this answer