(1D) Ordered Tokens: The Key to Unlocking Test-Time Scaling for Image Generation

TL;DR

The "scaling law" era is shifting from purely training-time compute to test-time compute. This paper proves that the way we discretize images (tokenization) determines whether a model can actually benefit from searching for better results at inference time. By moving from 2D spatial grids to 1D coarse-to-fine ordered tokens, the authors enable autoregressive models to be steered by verifiers, achieving massive quality gains and even training-free image generation.

Problem: The Blind Spots of Raster-Scan

Most current autoregressive (AR) image models, like LlamaGen or Janus, treat an image like a grid. They generate tokens in a "raster-scan" order (left-to-right, top-to-bottom).

The technical bottleneck here is verifiability. If you stop a 2D model halfway, the verifier (like CLIP) only sees the top half of a "ghostly" image or a few localized pixels. It has no idea if the entire composition is going to match the prompt. Consequently, search algorithms like Beam Search—which prune bad paths early—fail because the early scores are essentially noise.

The Insight: Coarse-to-Fine is Searchable

The authors hypothesize that 1D ordered tokens (specifically FlexTok) solve this. In this architecture, the first few tokens represent a low-resolution, global "sketch" of the whole image. As more tokens are added, the "sketch" is refined into a high-definition image.

Why does this matter?

1. Semantic Readouts: Even the first token carries a "global" category (e.g., "a bag" or "a plant").
2. Early Steering: A verifier can look at the 10th token and say, "This is looking like a cat, but I wanted a dog," and prune that branch immediately.

Figure 1: (a) 1D tokens provide interpretable intermediate readouts. (b) 1D tokens exhibit much better scaling behavior with test-time compute.

Methodology: The SoTo Framework

The researchers developed the Search-over-Tokens (SoTo) framework to systematically test three components:

• Search Algorithms: Comparing Best-of-N (sampling many, picking one), Beam Search (pruning partial paths), and Lookahead Search (rolling out to the end before scoring).
• Verifiers: Using models like CLIP, ImageReward, and even rule-based segmentors (Grounded-SAM) to grade the samples.
• AR Priors: Testing how much the generative model itself needs to know.

The Most Potent Setup: Beam Search + FlexTok

For 2D grid models, the only way to search is "Best-of-N" (randomly guessing many times), which is compute-inefficient. For 1D ordered tokens, Beam Search becomes the "gold standard." It allows the model to explore more effectively, finding high-quality "needles" in the token "haystack."

Figure 2: Overview of the SoTo framework: Algorithms, Verifiers, and Priors.

Breakthrough Results

1. Superior Test-Time Scaling (TTS)

As you increase the inference compute (NFE), the performance of 1D ordered models sky-rockets compared to 2D grid models. Specifically, a smaller 530M parameter model using search can outperform a 3.4B model that uses standard sampling.

Figure 3: 1D ordered tokens (FlexTok) derive significantly more value from Beam Search than the 2D baseline.

2. Training-Free Generation & Control

Possessing a "searchable" space is so powerful that the authors demonstrated zero-shot multimodal control. They could take a model trained only on text-to-image and force it to follow a reference image identity just by changing the verifier to a similarity model (DreamSim) during search.

Even more extreme: Generation without a Model. By performing Beam Search over a uniform distribution of tokens and using CLIP as a guide, they could generate recognizable images without using an AR prior at all—something impossible with 2D grids.

Critical Analysis & Conclusion

This work marks a pivot in generative AI. It suggests that our current obsession with larger spatial grids in VQ-VAEs might be hindering our ability to use inference-time compute.

Limitations:

• Verifier Hacking: If the search is too aggressive, it learns to "cheat" the verifier, producing images that the score likes but humans find weird.
• Detokenization Cost: In current implementations, decoding tokens into pixels for the verifier to "see" is a bottleneck.

Takeaway: To reach "Sora" or "GPT" levels of reasoning in vision, we need a hierarchy. 1D ordered tokens provide that hierarchy, making image generation as searchable—and thus as scalable—as a game of Chess or Go.