Three ways to turn a scan into features
The study compares three strategies for producing the features an AI model uses to make a decision from a brain scan. The first computes anatomical measurements directly: regional brain volumes, cortical surface areas and cortical thickness values derived with FreeSurfer's recon-all pipeline, 178 features in total, fed into a logistic regression. The second trains a convolutional neural network (ResNet-18 for classification, U-Net for segmentation) to learn features from images under supervision. The third pretrains a vision transformer on unlabeled scans by masked autoencoding, then finetunes it on each task.
The scale is what makes the comparison informative. The authors pool 18 publicly available datasets containing more than 96,000 MRI scans from roughly 80,000 participants, with splits made at the participant level so that all scans from one person stay in the same split. Seven tasks span Alzheimer's disease diagnosis, epilepsy, Parkinson's disease, identification of tumor IDH mutations, brain age prediction, and segmentation of brain tumors and stroke lesions. Alzheimer's diagnosis is additionally tested on seven external cohorts, which is where a simple model's robustness matters most.
A linear anatomical model holds its ground
The headline result is a near-tie. Across the evaluated diagnostic and segmentation tasks, the authors report no significant performance differences between pretrained vision transformers, CNNs and the anatomical logistic regression. The anatomical model is not a straw man: a supplementary comparison against a random forest on dementia detection, repeated at 10%, 30% and 100% of training data, showed only small differences, which is why logistic regression serves as the main anatomical baseline.
The task-level picture is mixed rather than uniformly flat. For Alzheimer's diagnosis, the ASP vision transformer achieved the highest macro-AUC in most cohorts, but statistical significance over the CNN and anatomical logistic regression was reached only for OASIS 2. For Parkinson's disease and epilepsy, the pretrained vision transformer reached the highest AUC, with a significant improvement only over BrainIAC. For IDH mutation identification the ASP transformer again had the highest AUC without statistical significance. In segmentation, U-Net had the best Dice score for adult gliomas and the pretrained transformer for stroke lesions, with differences significant only relative to UNETR. The CNN could not be applied to the epilepsy task because training data were too limited.
Deep models appear to rediscover anatomy
The authors' interpretation is that the two families are not doing fundamentally different things. CNNs and pretrained vision transformers learn features that implicitly capture relevant anatomical information, which would explain why explicit anatomical extraction adds no measurable diagnostic advantage on these tasks. The paper supports this with a probing experiment: during masked-autoencoder pretraining, linear probes on the transformer embeddings predict volumes of different brain regions with increasing accuracy as training proceeds, showing that anatomical information accumulates in the embedding even without being asked for.
That probing result also carries a caveat the authors state directly: the embeddings reflect anatomy only to some extent. Prediction accuracy varies by target, with larger, higher-contrast structures such as cortex volume easier to read out than smaller, less distinct regions such as the hippocampus, and binary gender easier to infer than age, which requires finer-grained regression. So the claim is not that learned features equal a full anatomical description, but that they capture enough of it for these clinical labels.
Anatomy-guided pretraining separates on brain age
The paper's own methodological contribution, Anatomy Segmentation Pretraining (ASP), starts from that probing observation and pushes further. After standard masked-autoencoder pretraining, a lightweight decoder is trained to segment 39 brain regions selected for disease relevance and anatomical scale, discarding regions contributing less than 1% of total brain volume and merging symmetric or related subunits. The decoder deliberately has no skip connections and is kept small, so that anatomical information is forced into the transformer embedding rather than into local decoder pathways. A brain-region prompt selects which region to segment, avoiding the memory cost of 39 separate 3D logit maps.
The payoff is concentrated in age estimation. ASP achieved a mean absolute error of 2.54 years on the internal test cohort and 4.19 years in both the NACC and ADNI external cohorts. The pretrained transformer without ASP was close internally, only 0.35 years worse, but the gap widened to more than one year externally. Logistic regression trailed the transformer approaches by 1 to 2 years while consistently beating the CNN. An ablation on the external NACC cohort showed that removing any element — prior MAE pretraining, initial encoder freezing, or alternating MAE epochs to prevent forgetting — reduced performance, as did training from scratch.
Age errors grow at the edges of the data
Two conditions bound the age result. Stratifying by sex and age left the model ranking stable, but age-based stratification showed larger errors for individuals over 80, a group less represented in the training and pretraining data. Separately, when the best model was applied to clinical populations, age estimates for dementia patients in NACC and ADNI came out consistently higher than their biological age across most age intervals. That is a property of the model's output on a clinical group, not evidence that the model detects disease; the paper reports the pattern without claiming a diagnostic use for it.
The comparison also depends on how the anatomical baseline is built. Its features come from FreeSurfer's recon-all workflow, which the authors note takes about 6.11 hours per 3D image on average, and which also served as the quality-control filter, excluding images that failed initial processing. The paper points out that faster deep-learning segmentation tools offer an efficient alternative to this extraction cost, so the practical trade-off between the two paradigms is compute, not accuracy, on the tasks studied.
The tie is scoped to these tasks and cohorts
The equivalence claim is bounded by what was tested: 3D structural T1-weighted MRI, seven clinical tasks, and publicly available cohorts with the preprocessing and participant-level splits described. The statistical picture is also softer than a single headline suggests — most task-level advantages for the transformer models were not significant, and where significance appeared it was often against one specific comparator rather than across the board. The epilepsy CNN result is a concrete reminder that data volume, not architecture, can decide which models are even runnable.
Read together, the results argue for matching model complexity to the question. For diagnostic classification and lesion segmentation on structural T1 scans, a linear model over 178 anatomical features is a legitimate baseline that complex frameworks did not significantly beat. The exception the authors establish is brain age estimation, where injecting anatomy into pretraining produced a measurable and externally replicated error reduction, and where the advantage over plain pretraining grew from 0.35 years internally to more than one year on external cohorts — the clearest signal in the study that anatomy-aware pretraining transfers.
Sources used in this answer
Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI
A linear model on 178 FreeSurfer anatomical features matched CNN and pretrained vision transformer performance across seven structural MRI tasks with no significant differences, while the authors' Anatomy Segmentation Pretraining reached a mean absolute error of 2.54 years internally and 4.19 years externally in brain age estimation.