Can large language models score chronic pain narratives reliably?

A rubric, not a word count

The study asked whether a language model can read what patients say about their pain and turn it into numbers that behave like clinical measurements. Participants were interviewed over Zoom, and the recordings were transcribed with Whisper and passed to Llama-3-405B-Instruct, run locally on IBM servers for HIPAA compliance. The model was prompted as a clinical evaluator using clinician-style rubric ratings, with temperature set to 0 and a 4095-token cap to make outputs reproducible.

It produced nine domain scores. Seven mirror constructs that questionnaires already measure: physical pain, quality of life, emotional pain, catastrophizing, depression, anxiety, and rumination. Two are new: agency deficit and narrative fragmentation. The distinction matters because the first group can be checked against existing instruments, while the second group has no questionnaire equivalent to check against.

The comparison that makes the method legible is against simple word counting. When the nine model scores were reduced to two dimensions, the three cohorts clustered separately; counting depression- and pain-related terms produced extensive overlap between groups. The authors read this as evidence that the model is responding to context rather than vocabulary frequency, and word error rate from transcription showed no meaningful within-cohort correlation with any model metric.

Three groups, one interview protocol

The cross-sectional study enrolled 131 people: 67 with chronic low-back pain, 33 with major depressive disorder, and 31 pain-free healthy controls, all older than 40 and screened to exclude cognitive impairment. Chronic low-back pain required more than one year of pain below the T-10 vertebra, no radiculopathy, and average intensity above 4 out of 10; those patients were excluded if depression severity exceeded 19 on the MADRS. Depression patients met DSM-V criteria with MADRS at or above 19 and were excluded for any current chronic pain.

The interview had a generic section and a condition-specific section. Except for the Reddit comparison, the reported model metrics came from the generic section, where patients were not specifically discussing their condition. That design choice matters: it means the scores were not simply picking up whether someone was talking about back pain.

Group patterns followed the selection criteria. Chronic low-back pain patients received the highest physical pain ratings, while depression patients received the highest ratings for impaired quality of life, negative affect, and agency deficit. On emotional pain and rumination, the back-pain group sat between depression patients and healthy controls, elevated relative to controls but below the depression group. Normalizing scores by interview word count produced the same pattern, as did adjustment for age and sex.

Most correlations survived correction, with a moderate median

Convergent validity here means agreement between model scores and instruments already used to measure the same construct. The authors correlated model metrics with numeric rating scales, visual analog scales, and self-administered questionnaires using Spearman's rho. Across both patient groups, 85 of 117 correlations, or 72.6%, survived false-discovery-rate correction, with a median rho of 0.40 and a range of 0.28 to 0.57.

In the depression group alone, where five clinical instruments were available, 27 of 45 correlations, or 60.0%, survived correction, with a higher median rho of 0.54 and a range of 0.39 to 0.73. The paper highlights that model-derived depression ratings in both patient groups correlated at rho above 0.45 with MADRS scores administered by a research assistant on the screening day, separate from the interview day. That separation in time is what the authors use to argue the agreement is not just same-session consistency.

A median rho near 0.40 is meaningful but not interchangeable agreement. The authors also checked whether broad correlations reflected overlap among the questionnaires themselves, and report that this was the case, for example with physical pain correlating with both the pain rating scale and a depression subscale at rho = 0.40. Readers should treat these numbers as evidence of relatedness between measures, not as proof that the model reproduces a questionnaire score.

Sparse links suggest distinct dimensions

If the nine metrics were mostly restating each other, the approach would add little beyond a single distress score. To test this, the authors estimated a network of direct and partial correlations among the metrics in the back-pain group using Graphical Lasso, a method that shrinks weak conditional dependencies to zero. In the partial correlation network, 38.9% of edges were set to zero, meaning many bivariate associations were mediated by other variables rather than reflecting direct links.

Robustness checks supported the structure. Across a range of regularization settings, sparsity stayed between 27.8% and 38.9%, and the number of significant edges stayed at 2 to 3 out of 36 possible pairs. An effective dimensionality analysis of the raw correlation matrix gave 4.9 to 7.2, compared with an expected 8.45 for random data of the same size, which the authors interpret as the variables corresponding to largely independent dimensions.

Hierarchical clustering of the partial correlation matrix produced two clusters in the back-pain group, which the authors describe as reminiscent of the emotional and cognitive-evaluative dimensions of chronic pain. The same analysis in the depression group produced three clusters while keeping a similar emotional-cognitive global structure. These cluster labels are the authors' interpretation of the network geometry, not a separately validated factor structure.

Agency deficit separates groups; fragmentation does not

The two novel metrics do not carry equal weight in the results. Agency deficit, the sense of diminished control over one's life and condition, was highest in depression patients and contributed strongly to classification. Combining agency deficit with rumination classified the three groups with an F1 of 0.851, and agency deficit with narrative fragmentation reached an F1 of 0.71, using metrics that were not part of the inclusion criteria.

Narrative fragmentation, which concerns how coherently a personal account is organized, behaved differently. It showed the smallest between-group differences, did not differ between patient groups, and was elevated in both relative to healthy controls. The authors suggest it may be a transdiagnostic marker of psychological distress, but note it correlated only moderately with pain intensity and was not associated with affective or functional measures. They state plainly that further validation is needed to establish its construct validity and clinical relevance.

That asymmetry is the practical takeaway for anyone weighing these constructs. Agency deficit has a group-level signal and a classification result behind it; narrative fragmentation currently has a group-level elevation against controls and an explicit call for more validation. Treating the two as equally established would overstate the evidence.

Online posts reproduce the pattern at higher levels

To test whether the characterization holds outside a clinical sample, the authors scored posts from two Reddit communities, r/chronicpain with 4902 posts and r/depressed with 2278, using the condition-specific interview section because those posts are condition-specific. The direction of the pattern matched the clinical cohorts: every metric was higher in r/depression than in r/chronicpain except physical pain, where the reverse held.

The levels did not match. Except for physical pain, which was similar between clinical and online cohorts, all metrics were higher in the online communities. The authors offer an interpretation rather than a demonstrated mechanism: assessments of diagnostically unverified individuals may be influenced by the context and style of text generation, potentially limiting validity outside professional settings. They call for validation in larger, independent, well-characterized clinical cohorts.

This comparison is best read as a consistency check on direction, not as evidence that the model scores Reddit users the way it scores patients. The higher absolute levels in online text are a warning about calibration across settings, and the paper does not resolve whether the cause is population, context, or writing style.

A research tool, not a clinical device

The authors are explicit about the boundary. Reproducibility and absolute calibration of model ratings remain open questions, particularly for metrics like narrative fragmentation that typically rely on human raters. They propose future work with human-rated exemplars to compare zero-shot and few-shot prompting, preregistered scoring rubrics, and prompt-formulation comparisons to test how sensitive agency and fragmentation ratings are.

Clinical deployment would require evaluation under applicable medical device regulations, prospective multicenter validation across diverse patient populations, formal assessment of clinical performance and potential bias in demographic and linguistic subgroups, cybersecurity evaluation, and human factors testing before regulatory approval. The paper therefore describes an automated framework that remains a research tool.

The sample also bounds the claims. With 67 back-pain patients, 33 depression patients, and 31 controls, all over 40 and screened for cognitive status, the results describe these well-characterized groups under a single interview protocol conducted by the same coordinator. The classification F1 values come from this sample and are not a deployment estimate.

Agreement, separation, and calibration are different claims

The paper's strongest claim is convergent validity: model scores correlate with established questionnaires, with 72.6% of correlations surviving correction at a median rho of 0.40. A separate claim is group separation: scores differ across chronic low-back pain, depression, and healthy controls in directions consistent with how the groups were selected, and the sparse network suggests the nine metrics are not redundant. A third claim, calibration, is the weakest, and the Reddit comparison is where it visibly strains.

For readers deciding what to take from this, the useful split is between constructs with questionnaire anchors and constructs without them. Physical pain, depression, anxiety, rumination, catastrophizing, emotional pain, and quality of life can be checked against instruments, and the reported correlations are the evidence to weigh. Agency deficit and narrative fragmentation cannot be checked that way yet; agency deficit has group separation and classification support, while narrative fragmentation has an elevated score in both patient groups against controls and an explicit need for further validation.

The priority the paper itself sets is validation in larger, independent, well-characterized cohorts before any clinical use, with particular attention to the human-rater-dependent metrics. Until then, the defensible reading is that interview-based language model scores carry real signal about pain and mood dimensions in this sample, and that the two novel constructs should be judged separately rather than as a package.

Sources used in this answer