How are Shapley value explanations used and misused in clinical prediction?

A Shapley value is relative to a reference population

A Shapley value explanation splits one model prediction into per-feature contributions. The paper's key point is that this split is not absolute: the contributions are typically expressed relative to the average prediction over a reference population chosen by the researcher. Change that population and every contribution can change.

The paper illustrates the size of this effect with a numerical example. A reference population of 100 individuals had an average prediction of 5.48, while the average prediction across all 1,000 individuals in the data set was 4.92. Had a different random sample been drawn with an average prediction of 5.00, all the Shapley value explanations would have shifted accordingly. Small reference samples therefore make explanations unstable, and larger samples reduce that instability.

The reference population can also be sampled selectively, for example including only women under 30, in which case the explanations become relative to a population with those specific characteristics. The paper treats this as a legitimate design choice, but one that must be stated, because it changes what the numbers mean.

Marginal and conditional Shapley values answer different questions

The paper distinguishes two common types of Shapley value explanation. Marginal, also called interventional, Shapley values help describe the model's functional form. Conditional Shapley values help describe correlations present in the development data, even if the model did not learn them, because they tend to spread credit among correlated features.

This distinction matters for interpretation: a conditional explanation can attribute contribution to a feature the model barely uses, simply because that feature is correlated with one the model does use. The paper notes that other types exist, including causal and baseline Shapley values, and that marginal and conditional explanations are generally recommended because they are more practically useful and involve fewer arbitrary researcher decisions. Baseline Shapley values, by contrast, require choosing a single set of baseline values, a choice that may feel arbitrary yet can substantially change the explanations.

The paper also notes that at least 24 distinct algorithms exist for computing Shapley value explanations, and that different methods may produce values with bias, introduced when assumptions are violated, or variance, due to stochastic processes in the computation.

One hundred papers drawn from a one-year window

The authors ran a scoping review of publications indexed in PubMed between July 2023 and July 2024. To be included, a paper had to be an English-language peer-reviewed journal article, identify the use of Shapley value explanations in its title or abstract, and use a supervised machine learning method to model a patient-related health outcome from tabular clinical features. Studies using exclusively non-clinical features such as genomics or radiomics were excluded.

Screening covered 3,876 unique abstracts, of which 515 met the inclusion criteria; the authors then drew a random sample of 100 publications for data extraction. Two evaluators independently screened and extracted data, resolving disagreements by discussion. For publications with more than one final model, the authors extracted data from all final models but reported one model per publication, choosing the model for the primary research question or, where aims were many and distinct, the best-performing model.

The sample spans clinical domains, with oncology the largest at 20 publications, followed by neurology at 13, surgery at 12, and cardiology at 10. Because the sample is random and limited to a one-year window, the percentages describe reporting patterns in that sample rather than a census of all clinical Shapley use.

Computation choices go unreported in most papers

The review's central empirical finding concerns disclosure. Of the 100 publications, 91% did not disclose the algorithm used to compute the Shapley value explanations, and 97% did not disclose the source or size of the reference population used in the calculations.

These two omissions are consequential rather than cosmetic. Without the algorithm, a reader cannot judge whether the values carry bias or variance from the estimation procedure. Without the reference population, a reader cannot know what the contributions are measured against, and cannot assess stability, since small reference samples can produce explanations that change substantially when recomputed with a different random sample.

The authors add a further layer of instability: prediction models themselves can be unstable when developed with small sample sizes, and that instability is inherited by the explanations. They cite recent work indicating that Shapley value explanations may be even more unstable than the predictions themselves. In the reviewed sample, the missing reporting means readers often cannot form an intuition for how stable the presented explanations are.

Trust and key-feature naming dominate stated motivations

The authors grouped author-reported motivations into four themes, which overlap within publications. Trustworthiness was the most common at 83%, followed by identification of key predictors at 77%, clinical decision support at 33%, and exploratory analyses at 12%.

For trustworthiness, the paper identifies five sub-themes that authors said helped elicit trust in their prediction model, including biological plausibility at 10% and consistency with existing literature at 30%. For key-feature identification, the authors note a wide vocabulary in use, with the same kind of output described variously as key predictors, risk factors, potential biomarkers, determinants, contributing factors, crucial factors, pivotal factors, influencing factors, and protective factors.

The paper's concern is that this vocabulary often carries more inferential weight than the explanation supports. Interpretations should strictly refer to the model's behavior or to associations in the training data set.

Causal language outruns what the explanations show

The review found that many publications lacked nuance when interpreting identified key features, making statements that were too general or using language that could be confused with causal associations. The paper gives a concrete rewrite: a statement that length of stay, triglycerides, and lactate dehydrogenase "had the greatest effect on 30-day unplanned readmission" would be more accurately phrased as having the greatest effect on the predicted 30-day unplanned readmission.

The gap widens for clinical decision support. Some authors described modifiable variables, such as intraoperative vital signs like blood pressure, as risk factors controllable through intervention during cardiac surgery; others proposed tools for patient counseling or suggested that actionable advice could follow from the explanations. The review found that none of the publications used an explicitly causal model, disclosed an underlying causal graph, or assessed other assumptions necessary for causal inference. The authors therefore conclude that suggestions for actionable intervention based solely on Shapley value explanations were generally unwarranted, and that plans for individualized treatment or new interventions should be reported as hypotheses.

The paper also notes a counterexample of careful phrasing, in which authors stated that shedding light on which features exert stronger influence on prediction does not necessarily suggest a true biologic relationship between features and outcome.

The reporting gap is the actionable target

The paper's practical recommendations follow directly from the two disclosure failures. Interpretations should be phrased within the context of the prediction model or the development data: "higher values of input X result in higher predicted risk of the outcome" rather than "X increases the risk of the outcome," and "in our sample of N retrospective observations, older individuals had higher prevalence of disease A" rather than "age influences the development of disease A."

The authors also recommend making the computation choices clear, for example by publishing code in an appendix or repository, which they describe as the easiest way to make the choices made during computation transparent.

The priority this paper sets is therefore not whether to use Shapley values but which details must accompany them. A clinical prediction paper that reports Shapley explanations without naming the algorithm and the reference population leaves readers unable to judge stability or meaning, and one that converts feature rankings into intervention advice without a causal model has moved beyond what the explanation can support. The distinction between describing a prediction and describing a disease is the line the review asks readers to hold.

Sources used in this answer