Can pixel-based language models predict human reading time and word skipping?

Why tokenization limits reading models

Computational psycholinguistics commonly estimates surprisal, the negative log probability of a word given its context, from neural language models and uses it to explain how long readers look at words. The paper argues that conventional token-based language models (TLMs) carry a structural bias here: their tokenizers and vocabularies are fixed in advance, without regard to cognitive plausibility. Because surprisal is computed over those predefined units, the choice of tokenizer can shift the estimate, and naive accumulation of subword probabilities can misallocate difficulty to a neighboring word.

A more fundamental limit is visual. Converting words to token IDs discards orthographic form, so a TLM cannot represent how text actually looks, including layout, font, or stroke complexity. That rules out studying bottom-up visual effects in reading, and it also excludes parafoveal information, the partial visual input readers get from words before fixating them. The paper frames this as a bottleneck for experimental design, not just a numerical nuisance.

Rendering text as image patches

Pixel-based language models (PLMs) take a different input. A rendering function converts a word sequence into an image, which is divided into non-overlapping patches; each patch is flattened and projected into an embedding, and the model is trained to reconstruct patches that were masked out. No tokenizer or discrete vocabulary is involved, so the model operates on the visual form of text directly.

The paper's key move is to turn this reconstruction ability into a reading measure. For a target word spanning several patches, the model masks those patches one at a time and records how badly it reconstructs the true pixels. Summing that reconstruction error over the word's patches gives visual word prediction difficulty (VWPD): a high value means the model struggled to predict the word's visual form from context, and the paper expects this to correspond to longer reading time.

Because off-the-shelf PLMs are bidirectional, hiding all future context would create a large gap between training and inference. The authors instead blur the future context, letting the model receive coarse cues such as approximate word length while withholding sharp detail. This is meant to simulate parafoveal sight, and the blurred version is the one used in the main experiments.

Two ways to score reconstruction error

The paper examines two reconstruction-error formulations. The first is instance-wise mean squared error in pixel space, the straightforward difference between predicted and actual patch pixels. The second borrows structural similarity (SSIM) from computer vision; since SSIM measures similarity, one minus SSIM serves as a difficulty score. The authors also compute a pixel-space entropy over predicted black-or-white pixel values, intended to capture uncertainty about the next word rather than noisy gray predictions.

These measures answer related but distinct questions. MSE and SSIM both quantify how hard the word's visual form was to reconstruct, while entropy quantifies uncertainty in the prediction. The paper reports them separately and does not treat them as interchangeable, which matters when comparing results across languages and measures.

Reading corpora and regression design

The evaluation uses naturalistic reading data in German, English, Spanish, Farsi, Korean, Simplified Chinese, Traditional Chinese, and Turkish, drawn from MECO Wave 1 or Wave 2 except for a Persian reading corpus. The outcome measures are first fixation duration, first-pass reading time, total reading time, first-pass skipping, and overall skipping. Durative measures exclude zero fixation durations and zero reading times; skipping is coded as a binary outcome, with first-pass skipping marking whether a region was skipped before a later region was fixated and overall skipping marking whether it was never fixated in the trial.

Each regression model includes one predictor of interest, either a PLM-derived measure or token-level surprisal or entropy from a comparable-size TLM, alongside controls for word length, word frequency, the previous word's length and frequency, and word position in the sentence. Random effects account for subjects and stimuli. Durative measures are log-transformed and skipping measures use a binomial family, fitted with generalized additive mixed models with all effects set as linear. The main comparisons are pixel difficulty without blurring, pixel difficulty with blurred future context, monolingual GPT-2 surprisal, multilingual GPT surprisal, and bidirectional BERT surprisal, plus combined and interaction models.

Pixel difficulty predicts gaze and skipping

The headline result is that VWPD computed in pixel space has strong predictive power for gaze duration and word skipping across the languages studied. The paper's summary states this directly and adds that the pixel-based advantage is complementary to existing token-level surprisal and entropy estimates, meaning the visual measure carries information the token measures do not.

The complementarity is tested through interaction models. When pixel difficulty and GPT surprisal are combined with an interaction, the paper reports gains over GPT surprisal alone, and an appendix figure describes this gain as suggesting that multimodal prediction encodes rich metacognitive knowledge about skipping. The cross-validation appendix reports the same tendencies: integrating token and visual prediction, especially with an interaction, improves reading-time and skipping modeling.

Advantages vary by script and measure

The paper's cross-lingual analyses with script variations are where the comparison becomes concrete. In the cross-validation results, visual prediction accounts for skipping better than token prediction in certain scripts, notably Persian, Korean, and Chinese. The main tables show the pattern is measure-specific rather than uniform: relative to GPT, pixel models with blurred context show positive log-likelihood differences for some skipping measures in some languages and negative ones in others, so the advantage depends on both the script and whether the outcome is a duration or a skipping decision.

The choice of reconstruction error also interacts with language. The cross-validation results do not contradict the observation that the MECO Chinese data show a stronger advantage for MSE-based VWPD, and the paper reports MSE and SSIM performance against GPT-2 separately by dataset. Whole-word masking, compared with blurring, produces a more nuanced pattern: an advantage for word skipping in Traditional Chinese but a disadvantage in Korean, which the authors read as a possible language-dependent subword processing strategy during naturalistic reading.

What the visual signal adds to expectation

The paper interprets its results as evidence for at least partial cognitive plausibility of PLMs, especially alignment with early-stage language processing such as first-pass skipping. The reasoning is that pixel-space prediction difficulty incorporates orthographic and parafoveal information that token-level surprisal cannot represent, so it captures a bottom-up visual contribution to reading that expectation-based token measures miss. The authors describe the work as the first to introduce a pixel-based language model into computational cognitive modeling.

That interpretation should be kept distinct from the measured result. The regressions show that pixel-derived measures add predictive power for the reading outcomes; the claim that this reflects incremental integration of bottom-up visual information into expectation-based reading is the authors' framing of what the pattern means. The paper's own discussion ties the result to prefixation processing on partial input, which is consistent with the blurred-future-context design but remains an interpretation rather than a direct manipulation of human parafoveal vision.

Where the claim holds and where it stops

The evidence is bounded by rendering and modeling choices. The authors use only bigram rendering, described as the most cross-lingually generalizable setting and the foundation of the only multilingual pixel-based model so far, and they acknowledge that visual word prediction metrics are highly sensitive to text rendering strategy; they call for future work on other rendering strategies and their effect on cognitive plausibility. For pixel blurring, they note that an intensive hyperparameter search was not conducted.

These limits matter for how the result should be used. The finding supports pixel-based prediction difficulty as a complement to token-level surprisal and entropy when modeling gaze duration and skipping, with the clearest script-specific skipping advantages in Persian, Korean, and Chinese in the cross-validation analysis. It does not establish that pixel measures should replace token measures, and the language- and measure-specific sign changes in the tables mean the practical priority is to test the interaction of visual and token predictors for the particular script and outcome at hand, rather than assuming a uniform gain.

Sources used in this answer