Do large language models share human sublexical sensitivity in pseudoword processing?

Pseudowords isolate form-based meaning cues

A pseudoword is a letter string that follows a language's orthographic rules but has no stored meaning. Because it has no lexical entry, a reader cannot fall back on memorized word knowledge; whatever meaning they infer must come from sublexical regularities, the statistical associations between small form units and meaning that the language already contains. That is why the paper treats pseudowords as probes: they strip away vocabulary and leave the form-to-meaning mapping exposed.

The study reuses two Italian experiments from Gatti et al. (2024). In each trial a target appears with two options, and the participant picks the one that feels more semantically similar. In Experiment 1 (EXP1) the target is a real Italian word and both options are pseudowords, one semantically related to the target (fastText cosine 0.20–0.44) and one unrelated (cosine ≤0.05). In Experiment 2 (EXP2) the target is a pseudoword and the options are one related and one unrelated real word. The two options in each trial were matched on orthographic length and Levenshtein distance, and in EXP2 also on word and lemma frequency, grammatical gender, number, and part of speech.

The human baseline came from 60 native Italian speakers per experiment, 50 trials each, with reaction times below 500 ms or above 50,000 ms excluded. That left 2,975 trials in EXP1 and 2,997 in EXP2. The key structural difference is what the real-word option does: in EXP2 it gives the model a familiar lexical anchor, while in EXP1 every option is an out-of-vocabulary string.

Alignment is trial-level agreement, not accuracy

The paper's main measure is human–model alignment: on each trial, does the model's answer match what an individual human participant chose? Each model ran three iterations per experiment per prompt condition, and the three responses were reduced to a single majority-vote answer. That answer was then compared with each participant's response, producing a binary aligned or misaligned outcome per participant–item pair. Alignment was estimated with an intercept-only binomial mixed model with crossed random intercepts for participant and item, tested against a 50% chance level.

This is not the same as asking whether a model is correct. There is no ground truth about which pseudoword is more semantically related; the benchmark is whether the model lands where humans land. The reference line is fastText, a character n-gram model that in prior work reliably predicts human pseudoword judgments. In this dataset human–fastText alignment was 66.3% in EXP1 and 58.1% in EXP2.

Two prompt conditions were tested. The baseline prompt gave only the task description. The role-based prompt added a persona of a native Italian speaker aged 19–34 with normal or corrected-to-normal vision, plus an instruction to use native-speaker intuition. GPT-4o, DeepSeek-V3, and DeepSeek-R1 were queried at temperature 0; o4-mini ran at its default because it does not support a temperature parameter; Minerva-7B ran locally with greedy decoding. Minerva-7B and DeepSeek-V3 produced some invalid responses, which were excluded, so their trial counts are lower than the full 2,975 and 2,997.

Real-word options lift alignment above chance

In EXP1, where both options were pseudowords, alignment ranged from 48.5% to 58.8% across all models and conditions. Only one result cleared chance: Minerva-7B under the baseline prompt at 58.8% (p=.018). Every other model and condition fell short of significance, and all LLMs sat well below the fastText reference of 66.3%.

EXP2 reversed the picture. With a real-word option available, four models reached above-chance alignment, and the best result slightly exceeded the fastText reference of 58.1%: GPT-4o with the role prompt at 59.7% (p=.002), DeepSeek-V3 baseline at 58.9% (p=.007), DeepSeek-R1 role at 58.3% (p=.009), and o4-mini at 57.8% (p=.016) and 58.5% (p=.008) in its two conditions. Minerva-7B, the Italian-first model, went the other way and sat at or below chance in EXP2 (44–51%).

That reversal raised a coverage concern, because Minerva-7B drew on fewer valid items. Restricting the models to items all five answered — 35 items and 2,082 trials in EXP1, 39 items and 2,337 trials in EXP2 — preserved the pattern: Minerva-7B approached but did not reach chance in EXP1 (57.2%, p=.052) and stayed at or below chance in EXP2 (44.1–49.5%), while the frontier models largely reproduced their EXP2 results. The persona manipulation itself had no significant pooled effect in either experiment (EXP1 β=−0.012, p=.621; EXP2 β=−0.015, p=.539).

The sublexical cosine cue does not transfer to LLMs

If LLMs were tracking the same form-based signal as humans, the item-level features that predict human–fastText agreement should also predict human–LLM agreement. The paper tested the same predictors Gatti et al. used: Dcos, the difference in fastText cosine similarity to the target; DL, the difference in string length; and DLD, the difference in Levenshtein edit distance. EXP2 added Dfreq and Dfreq_L, differences in log word and lemma frequency. Each predictor is the related option's value minus the unrelated option's value, so a positive value means the related option has the advantage on that dimension.

The result is mostly null. In EXP1 no model reached significance on any predictor. In EXP2 only DeepSeek-R1 showed a reliable effect, on Dcos (β=0.35, p=.008); all other models were non-significant. The human–fastText reference, by contrast, showed a reliable Dcos effect in both experiments (EXP1 β=0.20, p=.041; EXP2 β=0.27, p=.030).

The paper is careful about what that reference means. Because fastText picks whichever option is closer in cosine terms, the Dcos effect is expected by construction; it functions as a positive control showing the items carry a sublexical signal strong enough to be detected in this paradigm and tracked by humans. The null effects for the LLMs therefore point to insensitivity to that cue rather than to a weak stimulus set. The authors also note that Dcos is derived from fastText vectors for all models, so it indexes the sublexical semantic signal available in the stimuli, not a model-specific similarity measure.

Reasoning tokens do not track human difficulty

The third question asked whether items that are harder for humans also cost reasoning models more computation. Effort was operationalized as reasoning tokens consumed per trial, averaged across the three iterations; human effort was indexed by mean response time per item, with response entropy as a complementary measure. For each reasoning model and prompt condition, the paper correlated log mean reasoning tokens with mean log human reaction time across 50 items, then recomputed partial correlations controlling for Dcos to remove stimulus discriminability.

Only one association was reliable: o4-mini baseline in EXP1 (r=.28, p=.046; partial r=.29, p=.039), and it was positive, meaning items that took humans longer also drew more reasoning tokens. That survived the Dcos control. But no other model, condition, or experiment produced a consistent effect. On response entropy, only o4-mini in EXP2 showed reliable associations under both prompts (baseline r=.34, p=.015; role r=.29, p=.043), with more tokens for items where human responses were more evenly split. DeepSeek-R1 showed no significant correlation in either experiment.

The consistent positive direction for o4-mini is worth noting, but the paper does not treat it as a general finding. The associations did not replicate across experiments and were absent for DeepSeek-R1, so the results do not support a general correspondence between reasoning-token expenditure and human item-level processing demands on pseudoword tasks.

Tokenization and training coverage remain candidate explanations

The paper's interpretation is that LLMs do not necessarily share the sublexical cues that govern human pseudoword processing. It offers two candidate explanations rather than a settled mechanism. The first is tokenization: LLMs process text through subword units such as byte-pair encoding, which segments strings by frequency rather than by meaning-bearing form. For a pseudoword with no lexical entry, those subword tokens are the primary form-based cues available, and whether a tokenization isolates an Italian form–meaning cue is not guaranteed.

The second is training-data coverage. The observed alignment gap may reflect how much of the relevant language the model saw as much as how it processes sublexical structure. The paper states this account cannot be ruled out and suggests replication in higher-resource languages to disentangle the two. That limitation matters for how the results should be read: the study is a case study in Italian, and the comparison between an Italian-first model and English-centric frontier models is entangled with differences in training exposure.

A second limitation concerns sample size at the item level. The alignment analyses operated at the trial level with over 2,000 observations per fit, but the sublexical predictor models and the reasoning-token correlations aggregate to 50 items, so those estimates should be interpreted with caution. The trial-level alignment results are the firmer evidence; the item-level cue analyses are the more fragile ones.

The real-word cue is the paper's sharpest contrast

The most decision-relevant comparison in this paper is not a single accuracy number but the contrast between the two experiments. When every option was a pseudoword, the models mostly hovered at chance and trailed fastText badly. When a real word was available as an option, several frontier models reached or slightly exceeded the fastText reference. The paper reads this as evidence that LLMs lean on lexical familiarity rather than on the sublexical form–meaning regularities that humans use.

That reading has a concrete implication for anyone using LLMs as models of human pseudoword processing. Agreement on a task where a real word is present is weak evidence that the model shares the human mechanism, because the real word supplies an anchor the model can exploit. The cleaner test is the pseudoword-only condition, and there the models largely failed to beat chance. The sublexical predictor results reinforce this: the cue that reliably drove human–fastText agreement did not consistently drive human–LLM agreement, with DeepSeek-R1's EXP2 Dcos effect the lone exception.

The paper's own framing is appropriately narrow. It reports little of the sublexical sensitivity that governs human pseudoword processing, not none, and it leaves tokenization and training-data coverage as open candidates. The practical priority it supports is to treat real-word-anchored agreement and pseudoword-only agreement as different evidence, and to weight the pseudoword-only condition when judging whether a model tracks human form-based meaning cues.

Sources used in this answer