How reliable are the results from functional magnetic resonance imaging?

Functional magnetic resonance imaging (fMRI) is one of the most important methods for in vivo investigation of cognitive processes in the human brain. Within the last two decades, an explosion of research has emerged using fMRI, revealing the underpinnings of everything from motor and sensory processes to the foundations of social cognition. While these results have revealed the potential of neuroimaging, important questions regarding the reliability of these results remain unanswered. In this paper, we take a close look at what is currently known about the reliability of fMRI findings. First, we examine the many factors that influence the quality of acquired fMRI data. We also conduct a review of the existing literature to determine if some measure of agreement has emerged regarding the reliability of fMRI. Finally, we provide commentary on ways to improve fMRI reliability and what questions remain unanswered. Reliability is the foundation on which scientific investigation is based. How reliable are the results from fMRI?

[1]  J. Bartko.  The Intraclass Correlation Coefficient as a Measure of Reliability , 1966, Psychological reports.

[2]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[3]  J. Nunnally.  Introduction to Psychological Measurement , 1970 .

[4]  M. J. Safrit,et al.  Comparison of two nonparametric methods for estimating the reliability of motor performance tests. , 1977, Research quarterly.

[5]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[6]  D. Cicchetti,et al.  Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. , 1981, American journal of mental deficiency.

[7]  Jacob Cohen,et al.  Statistical Power Analysis For The Behavioral Sciences Revised Edition , 1987 .

[8]  R. Turner,et al.  Functional mapping of the human visual cortex at 4 and 1.5 tesla using deoxygenation contrast EPI , 1993, Magnetic resonance in medicine.

[9]  Ravi S. Menon,et al.  Functional brain mapping by blood oxygenation level-dependent contrast magnetic resonance imaging. A comparison of signal characteristics with a biophysical model. , 1993, Biophysical journal.

[10]  R. S. Hinks,et al.  Spin‐echo and gradient‐echo epi of human brain activation using bold contrast: A comparative study at 1.5 T , 1994, NMR in biomedicine.

[11]  Scott T. Grafton,et al.  Functional Mapping of Sequence Learning in Normal Humans , 1995, Journal of Cognitive Neuroscience.

[12]  R. Cox,et al.  Test-retest precision of functional MR in sensory and motor task activation. , 1996, AJNR. American journal of neuroradiology.

[13]  M Diemling,et al.  Reproducibility and postprocessing of gradient-echo functional MRI to improve localization of brain activity in the human visual cortex. , 1996, Magnetic resonance imaging.

[14]  J A Frank,et al.  Reproducibility of human 3D fMRI brain maps acquired during a motor task , 1996, Human brain mapping.

[15]  P. Vargha,et al.  A critical discussion of intraclass correlation coefficients. , 1997, Statistics in medicine.

[16]  S. Rombouts,et al.  Test-retest analysis with functional MR of the activated area in the human visual cortex. , 1997, AJNR. American journal of neuroradiology.

[17]  Jonathan D. Cohen,et al.  Reproducibility of fMRI Results across Four Institutions Using a Spatial Working Memory Task , 1998, NeuroImage.

[18]  N C Andreasen,et al.  Functional MRI statistical software packages: A comparative analysis , 1998, Human brain mapping.

[19]  Karen Faith Berman,et al.  Mapping Voxel-Based Statistical Power on Parametric Images , 1998, NeuroImage.

[20]  S. Rombouts,et al.  Within-subject reproducibility of visual activation patterns with functional magnetic resonance imaging using multislice echo planar imaging. , 1998, Magnetic resonance imaging.

[21]  M S Cohen,et al.  Stability, repeatability, and the expression of signal magnitude in functional magnetic resonance imaging , 1999, Journal of magnetic resonance imaging : JMRI.

[22]  Carol A. Seger,et al.  Striatal activation during acquisition of a cognitive skill. , 1999, Neuropsychology.

[23]  S. Strother,et al.  Reproducibility of BOLD‐based functional MRI obtained at 4 T , 1999, Human brain mapping.

[24]  N. Costes,et al.  Haemodynamic brain responses to acute pain in humans: sensory and attentional networks. , 1999, Brain : a journal of neurology.

[25]  P. Jezzard,et al.  Sources of distortion in functional MRI data , 1999, Human brain mapping.

[26]  A M Dale,et al.  Optimal experimental design for event‐related fMRI , 1999, Human brain mapping.

[27]  M R Symms,et al.  Reproducible localization of interictal epileptiform discharges using EEG-triggered fMRI. , 1999, Physics in medicine and biology.

[28]  Karl J. Friston,et al.  Variability in fMRI: An Examination of Intersession Differences , 2000, NeuroImage.

[29]  F Barkhof,et al.  fMRI of visual encoding: Reproducibility of activation , 1999, Human brain mapping.

[30]  T H Monk,et al.  CIRCADIAN RHYTHMS OF PERFORMANCE: NEW TRENDS , 2000, Chronobiology international.

[31]  Mark Hallett,et al.  The variability of serial fMRI data: Correlation between a visual and a motor task , 2000, Neuroreport.

[32]  L. K. Hansen,et al.  The Quantitative Evaluation of Functional Neuroimaging Experiments: The NPAIRS Data Analysis Framework , 2000, NeuroImage.

[33]  T. V. van Erp,et al.  Reproducibility of visual activation in functional MR imaging and effects of postprocessing. , 2000, AJNR. American journal of neuroradiology.

[34]  G L Wolford,et al.  Brain activations associated with shifts in response criterion on a recognition test. , 2001, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[35]  Ari Visa,et al.  Reproducibility of fMRI: Effect of the Use of Contextual Information , 2001, NeuroImage.

[36]  N. Andreasen,et al.  Anatomic and Functional Variability: The Effects of Filter Size in Group fMRI Data Analysis , 2001, NeuroImage.

[37]  G. Glover,et al.  Physiological noise in oxygenation‐sensitive magnetic resonance imaging , 2001, Magnetic resonance in medicine.

[38]  Karl J. Friston,et al.  Modelling Geometric Deformations in Epi Time Series , 2022 .

[39]  S. Rauch,et al.  Test-retest reliability of a functional MRI working memory paradigm in normal and schizophrenic subjects. , 2001, The American journal of psychiatry.

[40]  T. V. van Erp,et al.  Reproducibility of visual activation during checkerboard stimulation in functional magnetic resonance imaging at 4 Tesla. , 2001, Japanese journal of ophthalmology.

[41]  Stephen M. Smith,et al.  Temporal Autocorrelation in Univariate Linear Modeling of FMRI Data , 2001, NeuroImage.

[42]  F. Chollet,et al.  Within-Session and Between-Session Reproducibility of Cerebral Sensorimotor Activation: A Test–Retest Effect Evidenced with Functional Magnetic Resonance Imaging , 2001, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[43]  N. F. Ramsey,et al.  Reproducibility of fMRI-Determined Language Lateralization in Individual Subjects , 2002, Brain and Language.

[44]  Scott T. Grafton,et al.  Extensive Individual Differences in Brain Activations Associated with Episodic Retrieval are Reliable Over Time , 2002, Journal of Cognitive Neuroscience.

[45]  Guinevere F. Eden,et al.  Meta-Analysis of the Functional Neuroanatomy of Single-Word Reading: Method and Validation , 2002, NeuroImage.

[46]  Joseph A Maldjian,et al.  Multiple reproducibility indices for evaluation of cognitive functional MR imaging paradigms. , 2002, AJNR. American journal of neuroradiology.

[47]  Ranjan Maitra,et al.  Test‐retest reliability estimation of functional MRI data , 2002, Magnetic resonance in medicine.

[48]  Abraham Z Snyder,et al.  Reliability of functional localization using fMRI , 2003, NeuroImage.

[49]  Israel Liberzon,et al.  Habituation of Rostral Anterior Cingulate Cortex to Repeated Emotionally Salient Pictures , 2003, Neuropsychopharmacology.

[50]  N Jon Shah,et al.  Assessment of reliability in functional imaging studies , 2003, Journal of magnetic resonance imaging : JMRI.

[51]  J. Fell,et al.  Intrasubject reproducibility of presurgical language lateralization and mapping using fMRI , 2003, Neurology.

[52]  Thomas E. Nichols,et al.  Optimization of experimental design in fMRI: a general framework using a genetic algorithm , 2003, NeuroImage.

[53]  Gabriele Lohmann,et al.  Within-subject variability of BOLD response dynamics , 2003, NeuroImage.

[54]  K. Kiehl,et al.  Reproducibility of the hemodynamic response to auditory oddball stimuli: A six‐week test–retest study , 2003, Human brain mapping.

[55]  Peter Zhilkin,et al.  Affine registration: a comparison of several programs. , 2004, Magnetic resonance imaging.

[56]  Axel Schäfer,et al.  Hemodynamic Effects of Negative Emotional Pictures – A Test-Retest Analysis , 2004, Neuropsychobiology.

[57]  Charles R. G. Guttmann,et al.  Functional MRI of auditory verbal working memory: long-term reproducibility analysis , 2004, NeuroImage.

[58]  Lars Kai Hansen,et al.  Optimizing the fMRI data-processing pipeline using prediction and reproducibility performance metrics: I. A preliminary group analysis , 2004, NeuroImage.

[59]  Jing Z. Liu,et al.  Reproducibility of fMRI at 1.5 T in a strictly controlled motor task , 2004, Magnetic resonance in medicine.

[60]  Enrico Simonotto,et al.  Repeatability of motor and working-memory tasks in healthy older volunteers: assessment at functional MR imaging. , 2004, Radiology.

[61]  Stephen M Smith,et al.  Variability in fMRI: A re‐examination of inter‐session differences , 2005, Human brain mapping.

[62]  Marnie E. Shaw,et al.  How reliable are fMRI–EEG studies of epilepsy? A nonparametric approach to analysis validation and optimization , 2005, NeuroImage.

[63]  Tom Johnstone,et al.  Stability of amygdala BOLD response to fearful faces over multiple scan sessions , 2005, NeuroImage.

[64]  M. Manfredi,et al.  Long‐term Reproducibility of fMRI Activation in Epilepsy Patients with Fixation Off Sensitivity , 2005, Epilepsia.

[65]  Gregory G. Brown,et al.  Reproducibility of functional MR imaging: preliminary results of prospective multi-institutional study performed by Biomedical Informatics Research Network. , 2005, Radiology.

[66]  Richard J. Davidson,et al.  Comparison of fMRI motion correction software tools , 2005, NeuroImage.

[67]  P. Downing,et al.  Within‐subject reproducibility of category‐specific visual activation with functional MRI , 2005, Human brain mapping.

[68]  Lawrence L. Wald,et al.  Comparison of physiological noise at 1.5 T, 3 T and 7 T and optimization of fMRI acquisition parameters , 2005, NeuroImage.

[69]  Lukas Scheef,et al.  Functional 3.0-T MR assessment of higher cognitive function: are there advantages over 1.5-T imaging? , 2005, Radiology.

[70]  C. Dickey,et al.  Long-Term Reproducibility Analysis of Fmri using Hand Motor Task , 2005, The International journal of neuroscience.

[71]  Jesper Andersson,et al.  Valid conjunction inference with the minimum statistic , 2005, NeuroImage.

[72]  Andreas Schulze-Bonhage,et al.  The reliability of fMRI activations in the medial temporal lobes in a verbal episodic memory task , 2005, NeuroImage.

[73]  Jing Xu,et al.  Reproducibility of activation in Broca's area during covert generation of single words at high field: A single trial FMRI study at 4 T , 2006, NeuroImage.

[74]  J. Lancaster,et al.  Motivation and synthesis of the FIAC experiment: Reproducibility of fMRI results across expert analyses , 2006, Human brain mapping.

[75]  Russell A. Poldrack,et al.  Long-term test–retest reliability of functional MRI in a classification learning task , 2006, NeuroImage.

[76]  H. Brückmann,et al.  Reproducibility of activation in four motor paradigms. An fMRI study. , 2006, Journal of neurology.

[77]  Gary H. Glover,et al.  Reducing interscanner variability of activation in a multicenter fMRI study: Controlling for signal-to-fluctuation-noise-ratio (SFNR) differences , 2006, NeuroImage.

[78]  T. Salthouse,et al.  Short-term variability in cognitive performance and the calibration of longitudinal change. , 2006, The journals of gerontology. Series B, Psychological sciences and social sciences.

[79]  Diurnal changes of ERP response to sound stimuli of varying frequency in morning-type and evening-type subjects. , 2006, Journal of physiological anthropology.

[80]  S. MacDonald,et al.  Intra-individual variability in behavior: links to brain structure, neurotransmission and neuronal activity , 2006, Trends in Neurosciences.

[81]  M. Buonocore,et al.  Intrasubject reproducibility of functional MR imaging activation in language tasks. , 2006, AJNR. American journal of neuroradiology.

[82]  Stefan Knecht,et al.  The assessment of hemispheric lateralization in functional MRI—Robustness and reproducibility , 2006, NeuroImage.

[83]  Andrew P. Yonelinas,et al.  The intersubject and intrasubject reproducibility of FMRI activation during three encoding tasks: implications for clinical applications , 2006, Neuroradiology.

[84]  Anders M. Dale,et al.  Reliability in multi-site structural MRI studies: Effects of gradient non-linearity correction on phantom and human data , 2006, NeuroImage.

[85]  Benoit M. Dawant,et al.  Comparison of fMRI statistical software packages and strategies for analysis of images containing random and stimulus-correlated motion , 2007, Comput. Medical Imaging Graph..

[86]  Jong-Hwan Lee,et al.  REPRODUCIBILITY OF TRIAL-BASED FUNCTIONAL MRI ON MOTOR IMAGERY , 2007, The International journal of neuroscience.

[87]  Randy L. Gollub,et al.  Test–retest study of fMRI signal change evoked by electroacupuncture stimulation , 2007, NeuroImage.

[88]  Michael Borich,et al.  fMRI reliability in subjects with stroke , 2008, Experimental Brain Research.

[89]  Bradley R. Postle,et al.  Localization of load sensitivity of working memory storage: Quantitatively and qualitatively discrepant results yielded by single-subject and group-averaged approaches to fMRI group analysis , 2007, NeuroImage.

[90]  Neil Roberts,et al.  Activation of SI is modulated by attention: a random effects fMRI study using mechanical stimuli , 2007, Neuroreport.

[91]  Nick F. Ramsey,et al.  Test–retest reliability of fMRI activation during prosaccades and antisaccades , 2007, NeuroImage.

[92]  Karl J. Friston,et al.  Statistical parametric mapping , 2013 .

[93]  Richard Wise,et al.  A calibration method for quantitative BOLD fMRI based on hyperoxia , 2007, NeuroImage.

[94]  Richard B. Buxton,et al.  Reproducibility of BOLD, perfusion, and CMRO2 measurements with calibrated-BOLD fMRI , 2007, NeuroImage.

[95]  Kevin Murphy,et al.  How long to scan? The relationship between fMRI temporal signal to noise ratio and necessary scan duration , 2007, NeuroImage.

[96]  Yul-Wan Sung,et al.  Functional magnetic resonance imaging , 2004, Scholarpedia.

[97]  Steven L. Small,et al.  Test–retest reliability in fMRI of language: Group and task effects , 2007, Brain and Language.

[98]  J. Tonn,et al.  Reproducibility of Activations in Broca Area with Two Language Tasks: A Functional MR Imaging Study , 2007, American Journal of Neuroradiology.

[99]  Gary H Glover,et al.  Calibration of BOLD fMRI using breath holding reduces group variance during a cognitive task , 2007, Human brain mapping.

[100]  M. Brammer,et al.  Multisite fMRI reproducibility of a motor task using identical MR systems , 2007, Journal of magnetic resonance imaging : JMRI.

[101]  Andrew T. Smith,et al.  A comment on the severity of the effects of non-white noise in fMRI time-series , 2007, NeuroImage.

[102]  Gregory G. Brown,et al.  r Human Brain Mapping 29:958–972 (2008) r Test–Retest and Between-Site Reliability in a Multicenter fMRI Study , 2022 .

[103]  Massimo Filippi,et al.  Reproducibility of fMRI in the clinical setting: Implications for trial designs , 2008, NeuroImage.

[104]  Scott Holland,et al.  Reliability of fMRI for studies of language in post-stroke aphasia subjects , 2008, NeuroImage.

[105]  Arthur W. Toga,et al.  Provenance in neuroimaging , 2008, NeuroImage.

[106]  Jing Zhang,et al.  A Java-based fMRI Processing Pipeline Evaluation System for Assessment of Univariate General Linear Model and Multivariate Canonical Variate Analysis-based Pipelines , 2008, Neuroinformatics.

[107]  J. Theeuwes,et al.  Directing attention to a location in space results in retinotopic activation in primary visual cortex , 2008, Brain Research.

[108]  Izzie Jacques Namer,et al.  Test–retest reliability of a functional MRI anticipatory anxiety paradigm in healthy volunteers , 2008, Journal of magnetic resonance imaging : JMRI.

[109]  P. Hluštík,et al.  Effects of spatial smoothing on fMRI group inferences. , 2008, Magnetic resonance imaging.

[110]  Nick F. Ramsey,et al.  Within-subject variation in BOLD-fMRI signal changes across repeated measurements: Quantification and implications for sample size , 2008, NeuroImage.

[111]  Thomas E. Nichols,et al.  Power calculation for group fMRI studies accounting for arbitrary design and temporal autocorrelation , 2008, NeuroImage.

[112]  S. Lawrie,et al.  fMRI changes over time and reproducibility in unmedicated subjects at high genetic risk of schizophrenia , 2008, Psychological Medicine.

[113]  Steven C. R. Williams,et al.  Measuring fMRI reliability with the intra-class correlation coefficient , 2009, NeuroImage.

[114]  Thomas E. Nichols,et al.  Simple group fMRI modeling and inference , 2009, NeuroImage.

[115]  C. Keysers,et al.  Response to “ Voodoo Correlations in Social Neuroscience ” by Vul et al . – summary information for the press , 2009 .

[116]  Sylvie Belleville,et al.  Test–retest reliability of fMRI verbal episodic memory paradigms in healthy older adults and in persons with mild cognitive impairment , 2009, Human brain mapping.

[117]  Elliot T. Berkman,et al.  Correlations in Social Neuroscience Aren't Voodoo: Commentary on Vul et al. (2009) , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[118]  Andrea Sbarbati,et al.  Reproducibility of BOLD signal change induced by breath holding , 2009, NeuroImage.

[119]  Ranjan Maitra,et al.  Assessing certainty of activation or inactivation in test–retest fMRI studies , 2009, NeuroImage.

[120]  A. Toga,et al.  Multisite neuroimaging trials , 2009, Current opinion in neurology.

[121]  V. Glauche,et al.  Test–retest reliability of event-related functional MRI in a probabilistic reversal learning task , 2009, Psychiatry Research: Neuroimaging.

[122]  Elliot T. Berkman,et al.  Correlations in Social Neuroscience Aren't Voodoo , 2009 .

[123]  Cheng-Hung Chuang,et al.  Beyond p-values: averaged and reproducible evidence in fMRI experiments. , 2009, Psychophysiology.

[124]  John D. Van Horn,et al.  Unique and persistent individual patterns of brain activity across different memory retrieval tasks , 2009, NeuroImage.

[125]  K. Zilles,et al.  Coordinate‐based activation likelihood estimation meta‐analysis of neuroimaging data: A random‐effects approach based on empirical estimates of spatial uncertainty , 2009, Human brain mapping.

[126]  Sergi G. Costafreda,et al.  Pooling fMRI Data: Meta-Analysis, Mega-Analysis and Multi-Center Studies , 2009, Front. Neuroinform..

[127]  Joseph T. Devlin,et al.  Consistency and variability in functional localisers , 2009, NeuroImage.

[128]  B. Biswal,et al.  The resting brain: unconstrained yet reliable. , 2009, Cerebral cortex.

[129]  H. Pashler,et al.  Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition 1 , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[130]  Makoto Takahashi,et al.  Neural bases of goal-directed implicit learning , 2009, NeuroImage.

[131]  H. Abdi.  The General Linear Model , 2009 .

[132]  Michael B. Miller,et al.  The principled control of false positives in neuroimaging. , 2009, Social cognitive and affective neuroscience.

[133]  Stephen C. Strother,et al.  Evaluation and optimization of fMRI single-subject processing pipelines with NPAIRS and second-level CVA. , 2009, Magnetic resonance imaging.

[134]  Ian Marshall,et al.  Functional Magnetic Resonance Imaging (fMRI) reproducibility and variance components across visits and scanning sites with a finger tapping task , 2010, NeuroImage.

[135]  O. Dietrich,et al.  Test–retest reproducibility of the default‐mode network in healthy individuals , 2009, Human brain mapping.

[136]  L. Shah,et al.  Functional magnetic resonance imaging. , 2010, Seminars in roentgenology.

[137]  W. K. Simmons,et al.  The Selectivity and Functional Connectivity of the Anterior Temporal Lobes , 2009, Cerebral cortex.