Does Language-Model Questionnaire Scoring Reflect the Person or the Wording?

Wording alone can produce convincing agreement

A questionnaire is a list of sentences that becomes a map of a mind only through a person's answers. Language-based assessment reverses that arrangement: sentence embeddings, entailment models and large language models score how closely interviews, therapy dialogue or online posts match the item texts, and researchers then sum those scores into scales and correlate them into networks as though they were questionnaire answers.

The paper's central warning is that similarly worded items attract the same speech whatever the speaker feels. When cosine scorers embed speech units and item texts in one space and take inner products, the item activity profiles have covariance A⊤ΣA; if speech carries no item-specific signal, the geometry reduces to the item-text similarity matrix M. A model can therefore inherit a coherent, replicable map of symptoms before it has located anyone on that map.

One geometry per source makes the comparison possible

Psychometric representational similarity analysis places every source of psychometric information in a common format. The 25 PHQ-8 and PCL-C items become the conditions, and each source supplies an item-by-item geometry: co-endorsement across people for self-report, co-activation by speech for language, related meanings for item wording, and shared instrument clusters for theory. Each yields a representational dissimilarity matrix, Djk = 1 − rjk over item pairs, and correlating one matrix's entries with another's compares the sources.

The framework is implemented alongside person-level construct scoring in COMPASS 2.0. Speech is divided into units, every unit is scored against every item, and the same item activity profiles are read two ways: as a geometry, and as keyed construct scores that weight the profiles by an instrument's scoring key. The two read-outs answer different questions, and the paper keeps them separate throughout.

Language geometry followed wording more than self-report

For the 25 PHQ-8 and PCL-C items, embedding-scored speech geometry was nearly identical across the two interview samples (ρ = 0.92) and closer to wording (0.74, 0.72) than to self-report (0.43, 0.32). Self-report agreed across samples (0.56) and partly with wording (0.50, 0.35). Entailment geometry followed wording less (0.29) but agreed across samples (0.78) and partly with its own probe confusions (0.34, 0.36). The symptom-cluster theory model related weakly to every source (ρ ≤ 0.31).

Split-half reliability bounds put these numbers in context. Reliability was 0.95 for embedding-scored speech and 0.68 and 0.46 for self-report, so language reached about half its attainable agreement with self-report (0.53 after correction) and three quarters with wording (0.75). The paper notes that high reproducibility makes the language geometry's source especially consequential: a structure can be stable across populations because the measurement procedure carries it from one population to the next.

Registered tests found no structure beyond wording

The pre-registered discovery and confirmation analyses used clinical interviews from 275 participants in total: 189 DAIC-WOZ interviews with transcripts and 86 E-DAIC interviews with automatic transcripts. In both interview samples and both embedding models, geometry followed wording more closely than self-report, and no scorer showed registered evidence of population structure beyond wording.

A small exploratory entailment effect disappeared under another reference. Sensitivity analyses that whitened the population geometry by the pooled within-speaker covariance instead of M produced partial correlations with self-report that were small and mostly non-significant, and the paper states this reference is valid as a test of person structure only if within-speaker variation is nuisance with the same shape as between-speaker nuisance. Whitening is used as a diagnostic of departure from a reference, not as a recovery of personal structure.

Geometric agreement ignores who said what

Reassigning self-reports between participants left geometric agreement exactly unchanged — discovery entailment stayed at 0.33 before and after shuffling — while paired measures fell to their null. General-factor agreement fell from 0.49 to a shuffled range of −0.15 to 0.14, and COMPASS PHQ-8 agreement with self-reported totals fell from 0.54 to that same range; confirmation values before shuffling were 0.60 and 0.57.

This invariance follows from the comparison itself. Reordering people leaves the correlations among questionnaire items unchanged, so a population representational dissimilarity matrix retains no record of which report belongs to which interview. Geometry can establish that language organizes items as self-report does; establishing that it measures the right person requires paired read-outs.

Paired scores tracked distress more than specific symptoms

Exploratory analyses compared the first principal components of language-derived and self-reported items across participants, then examined item-specific correspondence after removing the self-reported general factor. For entailment, these components correlated r = 0.49 (95% CI 0.38 to 0.59) in discovery and 0.60 (0.44 to 0.72) in confirmation. Mean language-derived item scores correlated 0.53 and 0.58 with PHQ-8 totals. Cosine scorers carried less general-factor information (bge 0.26, 0.52; MiniLM 0.07, 0.33).

Entailment's association persisted beyond speech length and questionnaire-topic activation: after partialling answer-window count, word count and topic activation, COMPASS PHQ-8 scores correlated 0.52 (95% CI 0.43 to 0.61) and 0.49 (0.33 to 0.63) with self-reported PHQ-8 totals, while cosine-score correlations shrank. Item-specific convergence was much smaller, with same-item minus different-item correlations of 0.005 to 0.073. The paper describes these components as sample-specific summaries, not independently validated general factors, and notes that the differently scaled statistics do not partition symptom information.

Alliance scores behaved like relationship talk

Complementary analyses moved beyond symptoms. Earlier keyed Bond scores tracked relationship talk: across 784 sessions, the keyed client Bond score related to relationship-topic activation with R² = 0.91, and therapist-minus-client keyed Bond related to the therapist-minus-client topic gap at r = 0.94 across 783 sessions. Stance scoring distinguished counselling quality in a second corpus, but topic measures performed better in discovery and stance's advantage was not robust.

Wording structure was also examined across 34 instruments and the Research Domain Criteria framework, where wording clustered by construct domain. The paper's reading is that these controls are relevant wherever item texts probe language, and that keyed cosine sums should be interpreted as topic measures unless the scorer distinguishes stance.

What this changes for language-based assessment

The paper's practical priority is a wording baseline for any language-derived item structure. Symptom networks, factor structures and profiles, increasingly preferred to sum scores, require reporting geometric agreement with item texts, centring activities on eliciting questions, and testing residual structure against a declared reference. Alternatives such as within-speaker covariance should be examined under their stated assumptions, and person-level read-outs should be paired with person shuffles.

The most decision-relevant split in this study is between the two layers, not between models. Population geometry answers how an instrument organizes its subject matter and deliberately discards person identity; paired scores answer whether the right person was measured, and here they tracked general distress while item-specific convergence stayed between 0.005 and 0.073. The limits are specific: samples were modest (189 and 86 participants), with one interview each and short answers (median 12 and 8 words per window), automatic confirmation transcripts add error, registered-test power was high only for strong structure (94% at g = 0.4, 15% at g = 0.2 for 189 simulated people), and about a fifth of windows contained only one or two words scored without their question. Alliance analyses used demonstration-session quality rather than alliance ratings. The paper's own next test is the sharpest one: vary probe wording while holding a person's language fixed, then vary the speaker while holding the probes fixed, to see which features travel with the instrument and which travel with the person.

Sources used in this answer