Can vision transformers learn word meaning from images alone without language supervision?

Word meaning tested without any language labels

The paper asks whether a vision model can learn what words mean from pixels alone. The authors take fourteen models and show them images of words rendered on photographic backgrounds, then check whether the resulting representations place semantically related words near each other. The key models are DINOv3-H and DINOv3-7B, which were pretrained by self-supervision on images only, without paired captions or class labels.

The comparison set matters for reading the result. It includes smaller DINOv3 models, three DINOv2 models, three MAE models trained by masked reconstruction, Web-DINO-1B, two language-supervised models (CLIP-B and SigLIP2-B), and an untrained ViT-B that serves as a baseline for visual form alone. If an untrained network already grouped words by spelling, semantic structure would be unremarkable; the untrained baseline is there to test that.

Subtracting scrambled letters isolates the word signal

For each word image, the authors build a matched control image. The background, bounding box, position, color, and backing panel stay the same, but the letters' opacity mask is replaced by a phase-scrambled version. A word representation is then the difference between the model's embedding of the intact word image and its embedding of the control. This subtraction is meant to remove the background and layout while leaving the contribution of readable lettering.

The vocabulary contains 1,481 words and names plus 80 pseudowords, all lowercase and three to ten letters long, rendered across 96 text-free ImageNet backgrounds with 24 typography settings. Each word appears on four backgrounds, giving 299,808 images in total. Before centering, the vocabulary mean accounts for a median of 88% of the mean squared norm of the uncentered difference vectors across the fourteen models, so the authors subtract that mean to obtain a word direction. A word's direction therefore depends on the rest of the vocabulary, which is why the same word can have different directions in the lexical and class-name vocabularies.

Category and country links beat synonymy

The strongest evidence is neighbor structure. Brand names from the same industry group together, and nouns from the same semantic domain group together, measured as the fraction of a word's ten nearest neighbors that share its category or industry. Country-capital pairs such as Germany and Berlin show consistent vector offsets, similar to word embeddings trained on text. These results are reported for DINOv3-H and DINOv3-7B, which the authors describe as, to their knowledge, the first self-supervised vision models to show substantial semantic structure for rendered words.

Synonymy is weaker. In triplet tests, a synonym must be closer to an anchor than a named competitor. The one-edit condition makes this hard on purpose: the competitor differs from the anchor by a single letter insertion, deletion, or substitution, so spelling similarity competes directly with meaning. The paper's own discussion states that category membership and country-capital associations are more clearly expressed than synonymy, and that similarity and association are distinct, so factual relationships do not imply equally strong synonym knowledge.

A separate test asks whether a rendered class name sits near images of the object it names. The class-name vocabulary has 200 single-word ImageNet-1k class names, and each name's word direction is compared with a visual direction computed from 49,904 validation images not used as backgrounds. This connects word meaning to depicted object categories rather than only to other words.

Shared structure with text and speech encoders

The authors compare word directions from the vision models with representations of the same words from three Qwen3 text encoders and the w2v-BERT 2.0 speech encoder. Text vectors are built by placing each word in the same 96 neutral carrier sentences and averaging; speech vectors come from four carrier sentences spoken in eight voices at three rates, pooling only the frames where the word is spoken. Agreement is measured two ways: mutual 10-NN alignment, the average fraction of each word's ten nearest neighbors shared by two encoders, and linear CKA, which compares the full pattern of pairwise similarities.

The comparison shows shared structure among representations of the same words across modalities. The paper also reports a cross-encoder null that controls for word length but not all shared properties of word form, which limits how far claims of representational convergence can be pushed.

Finetuning helps DINOv3-H most

The finetuning experiment asks whether language supervision can build on word relationships the model already has. Self-supervised vision models are trained to match COCO images to either written captions or synthesized spoken captions, using a learned linear projection onto a frozen text or speech encoder and a symmetric InfoNCE loss. Images with detected text are excluded so the model cannot simply read writing in the frame. The projection is discarded afterward, so only the vision model's weights change.

Contrastive alignment with either encoder strengthened DINOv3-H's word-level semantic structure, while the other finetuned vision models were largely unchanged. After text finetuning, just four rendered images per word produced higher noun purity and synonym top-10 accuracy than all 96 images before finetuning. The authors read this as consistent with partial functional dormancy: caption supervision makes existing lexical structure easier to recover. They are explicit that the results do not distinguish eliciting existing relations from learning new ones.

Where the semantic structure could come from

The source of this structure remains unresolved, and this is the paper's central open question. Words could acquire associations from surrounding visual content, or from text embedded in the training frames. The latter could supply linguistic co-occurrence signals even for relations not directly depicted in a scene. The authors note that DINOv3's images are drawn mainly from public Instagram posts, so image-only training need not mean learning exclusively from depicted concrete objects.

The scope is also bounded in specific ways. The findings do not establish taxonomic hierarchies or sentence-level composition. Consistent country-capital offsets do not improve analogy retrieval over the country-only baseline, which the authors use to separate geometric regularity from analogical reasoning. A clue-combination test found that only CLIP-B showed contributions from both a broad and a narrow clue after correction across the thirteen pretrained models; DINOv3-H and DINOv3-7B showed a narrow-clue gain but no established broad-clue contribution, and the tests did not establish that these gains exceed the untrained ViT-B's.

Why rendered text matters for vision systems

The practical point is narrower than a claim that vision models understand language. The paper shows that the absence of language supervision does not guarantee that written text is inert to a system built on a vision encoder. Image-only pretraining can produce lexical and factual associations, and later finetuning toward language can strengthen some of them. The authors connect this to the observation that harmful requests rendered in images can bypass safeguards in vision-language systems, while stating that their experiments establish neither instruction following nor harmful behavior.

The priority the evidence supports is a comparison, not a verdict. When judging an image-only encoder, the question to ask is which relations it encodes: category membership and country-capital associations are the clear cases here, synonymy is the weak one, and the origin of the structure is still undetermined. That split, rather than a blanket claim about learning meaning from pixels, is what the paper's measurements actually support.

Sources used in this answer