Robust is not necessarily reliable: From within-subjects fMRI contrasts to between-subjects comparisons

&NA; Advances in cognitive and affective neuroscience come largely from within‐subjects comparisons, in which the functional significance of neural activity is determined by contrasting two or more experimental conditions. Clinical and social neuroscience studies have attempted to leverage between‐subject variability in such condition differences to better understand psychopathology and other individual differences. Shifting from within‐to between‐subjects comparisons requires that measures have adequate internal consistency to function as individual difference variables. This is particularly relevant for difference scores—which have lower reliability. The field has assumed reasonable internal consistency of neural measures based on consistent findings across studies (i.e., if a within‐subject difference in neural activity is robust, then it must be reliable). Using one of the most common fMRI paradigms in the clinical neuroscience literature (i.e., a face‐ and shape‐matching task), in a large sample of adolescents (N = 139) we replicate a robust finding: amygdala activation is greater for faces than shapes. Moreover, we demonstrate that the internal consistency of the amygdala in face and shape blocks was excellent (Spearman‐Brown corrected reliability [SB] > .94). However, the internal consistency of the activation difference between faces and shapes was nearly zero (SB = −.06). This reflected the fact that the amygdala response to faces and shapes was highly correlated (r = .97) across individuals. Increased neural activation to faces versus shapes could not possibly function as an individual difference measure in these data—illustrating how neural activation can be robust within subjects, but unreliable as an individual difference measure. Strong and reproducible condition differences in neural activity are not necessarily well‐suited for individual differences research—and neuroimaging studies should always report the internal consistency of, and correlations between, activations used in individual differences research. HighlightsRobust within‐subject differences are often used as individual difference measures.A robust within‐subject difference may not be internally consistent.Difference scores are problematic when constituent measures are highly correlated.Subtraction‐based measures should report the correlation between the two variables.Internal consistencies should be reported for difference and constituent scores