Influences on the Test–Retest Reliability of Functional Connectivity MRI and its Relationship with Behavioral Utility

Abstract Best practices are currently being developed for the acquisition and processing of resting‐state magnetic resonance imaging data used to estimate brain functional organization—or “functional connectivity.” Standards have been proposed based on test‐retest reliability, but open questions remain. These include how amount of data per subject influences whole‐brain reliability, the influence of increasing runs versus sessions, the spatial distribution of reliability, the reliability of multivariate methods, and, crucially, how reliability maps onto prediction of behavior. We collected a dataset of 12 extensively sampled individuals (144 min data each across 2 identically configured scanners) to assess test‐retest reliability of whole‐brain connectivity within the generalizability theory framework. We used Human Connectome Project data to replicate these analyses and relate reliability to behavioral prediction. Overall, the historical 5‐min scan produced poor reliability averaged across connections. Increasing the number of sessions was more beneficial than increasing runs. Reliability was lowest for subcortical connections and highest for within‐network cortical connections. Multivariate reliability was greater than univariate. Finally, reliability could not be used to improve prediction; these findings are among the first to underscore this distinction for functional connectivity. A comprehensive understanding of test‐retest reliability, including its limitations, supports the development of best practices in the field.

[1]  M. Raichle.  The brain's default mode network. , 2015, Annual review of neuroscience.

[2]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[3]  Timothy S. Coalson,et al.  A Surface-Based Analysis of Hemispheric Asymmetries and Folding of Cerebral Cortex in Term-Born Human Infants , 2010, The Journal of Neuroscience.

[4]  Evan M. Gordon,et al.  Individual-specific features of brain systems identified with resting state functional correlations , 2017, NeuroImage.

[5]  David A. Leopold,et al.  Dynamic functional connectivity: Promise, issues, and interpretations , 2013, NeuroImage.

[6]  Mario Pannunzi,et al.  Resting-state fMRI correlations: From link-wise unreliability to whole brain stability , 2016, NeuroImage.

[7]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[8]  Mark Jenkinson,et al.  The minimal preprocessing pipelines for the Human Connectome Project , 2013, NeuroImage.

[9]  M. Greicius,et al.  Decoding subject-driven cognitive states with whole-brain connectivity patterns. , 2012, Cerebral cortex.

[10]  J. Polimeni,et al.  96‐Channel receive‐only head coil for 3 Tesla: Design optimization and evaluation , 2009, Magnetic resonance in medicine.

[11]  Dustin Scheinost,et al.  The impact of image smoothness on intrinsic functional connectivity and head motion confounds , 2014, NeuroImage.

[12]  J. Ford,et al.  Default mode network activity and connectivity in psychopathology. , 2012, Annual review of clinical psychology.

[13]  Dustin Scheinost,et al.  Using connectome-based predictive modeling to predict individual behavior from brain connectivity , 2017, Nature Protocols.

[14]  X. Zuo,et al.  Test-retest reliabilities of resting-state FMRI measurements in human brain functional connectomics: A systems neuroscience perspective , 2014, Neuroscience & Biobehavioral Reviews.

[15]  Theo G. M. van Erp,et al.  Multisite reliability of MR-based functional connectivity , 2017, NeuroImage.

[16]  Jonathan D. Power,et al.  Recent progress and outstanding issues in motion correction in resting state fMRI , 2015, NeuroImage.

[17]  Martin A. Lindquist,et al.  Shrinkage prediction of seed-voxel brain connectivity using resting state fMRI , 2014, NeuroImage.

[18]  B. Biswal,et al.  The resting brain: unconstrained yet reliable. , 2009, Cerebral cortex.

[19]  Gustavo Deco,et al.  Can sliding-window correlations reveal dynamic functional connectivity in resting-state fMRI? , 2016, NeuroImage.

[20]  Bernard Ng,et al.  Optimization of rs-fMRI Pre-processing for Enhanced Signal-Noise Separation, Test-Retest Reliability, and Group Discrimination , 2015, NeuroImage.

[21]  John D. Storey.  A direct approach to false discovery rates , 2002 .

[22]  Satrajit S. Ghosh,et al.  The Healthy Brain Network Serial Scanning Initiative: a resource for evaluating inter-individual differences and their reliabilities across scan conditions and sessions , 2016, bioRxiv.

[23]  Elizabeth Gilbert,et al.  Reproducibility Project: Results (Part of symposium called "The Reproducibility Project: Estimating the Reproducibility of Psychological Science") , 2014 .

[24]  Gary H. Glover,et al.  Reducing inter-scanner variability of activation in a multicenter fMRI study: Role of smoothness equalization , 2006, NeuroImage.

[25]  Edward H. Haertel,et al.  4 Reliability Coefficients and Generalizability Theory , 2006 .

[26]  L. Cronbach.  Five perspectives on the validity argument. , 1988 .

[27]  G. Northoff.  Are Auditory Hallucinations Related to the Brain's Resting State Activity? A 'Neurophenomenal Resting State Hypothesis' , 2014, Clinical psychopharmacology and neuroscience : the official scientific journal of the Korean College of Neuropsychopharmacology.

[28]  S. Debener,et al.  Default-mode brain dysfunction in mental disorders: A systematic review , 2009, Neuroscience & Biobehavioral Reviews.

[29]  Xi-Nian Zuo,et al.  Short-term test–retest reliability of resting state fMRI metrics in children with and without attention-deficit/hyperactivity disorder , 2015, Developmental Cognitive Neuroscience.

[30]  L. Shah,et al.  Reliability and reproducibility of individual differences in functional connectivity acquired during task and resting state , 2016, Brain and behavior.

[31]  D. Yurgelun-Todd,et al.  Reproducibility of Single-Subject Functional Connectivity Measurements , 2011, American Journal of Neuroradiology.

[32]  Thomas E. Nichols,et al.  Scanning the horizon: towards transparent and reproducible neuroimaging research , 2016, Nature Reviews Neuroscience.

[33]  Dustin Scheinost,et al.  Alterations in Anatomical Covariance in the Prematurely Born , 2015, Cerebral cortex.

[34]  B. Leventhal,et al.  Unraveling the Miswired Connectome: A Developmental Perspective , 2014, Neuron.

[35]  Lucina Q. Uddin,et al.  The anterior insula in autism: Under-connected and under-examined , 2009, Neuroscience & Biobehavioral Reviews.

[36]  Evan M. Gordon,et al.  Functional System and Areal Organization of a Highly Sampled Individual Human Brain , 2015, Neuron.

[37]  Simon B. Eickhoff,et al.  An improved framework for confound regression and filtering for control of motion artifact in the preprocessing of resting-state functional connectivity data , 2013, NeuroImage.

[38]  A. Hayes.  Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach , 2013 .

[39]  Simon B. Eickhoff,et al.  Resting-state test–retest reliability of a priori defined canonical networks over different preprocessing steps , 2017, Brain Structure and Function.

[40]  P. Vargha,et al.  A critical discussion of intraclass correlation coefficients. , 1997, Statistics in medicine.

[41]  Timothy O. Laumann,et al.  Methods to detect, characterize, and remove motion artifact in resting state fMRI , 2014, NeuroImage.

[42]  M. Chun,et al.  A neuromarker of sustained attention from whole-brain functional connectivity , 2015, Nature Neuroscience.

[43]  Archana Venkataraman,et al.  Intrinsic functional connectivity as a tool for human connectomics: theory, properties, and optimization. , 2010, Journal of neurophysiology.

[44]  M. Baker.  1,500 scientists lift the lid on reproducibility , 2016, Nature.

[45]  J. Andrews-Hanna,et al.  Large-Scale Network Dysfunction in Major Depressive Disorder: A Meta-analysis of Resting-State Functional Connectivity. , 2015, JAMA psychiatry.

[46]  M. B. Nebel,et al.  Quantifying the reliability of image replication studies: The image intraclass correlation coefficient (I2C2) , 2013, Cognitive, affective & behavioral neuroscience.

[47]  Dustin Scheinost,et al.  Disruption of Functional Networks in Dyslexia: A Whole-Brain, Data-Driven Analysis of Connectivity , 2014, Biological Psychiatry.

[48]  Xenophon Papademetris,et al.  Groupwise whole-brain parcellation from resting-state fMRI data for network node identification , 2013, NeuroImage.

[49]  Thomas E. Nichols,et al.  A positive-negative mode of population covariation links brain connectivity, demographics and behavior , 2015, Nature Neuroscience.

[50]  A. Anderson,et al.  Respiratory effects in human functional magnetic resonance imaging due to bulk susceptibility changes. , 2001, Physics in medicine and biology.

[51]  Kuncheng Li,et al.  Reliability correction for functional connectivity: Theory and implementation , 2015, Human brain mapping.

[52]  Ann S. Choe,et al.  Reproducibility and Temporal Structure in Weekly Resting-State fMRI over a Period of 3.5 Years , 2015, PloS one.

[53]  H. Pashler,et al.  Editors’ Introduction to the Special Section on Replicability in Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[54]  Stephen M. Smith,et al.  The future of FMRI connectivity , 2012, NeuroImage.

[55]  Ann K. Shinn,et al.  Default mode network abnormalities in bipolar disorder and schizophrenia , 2010, Psychiatry Research: Neuroimaging.

[56]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[57]  Evan M. Gordon,et al.  On the Stability of BOLD fMRI Correlations , 2016, Cerebral cortex.

[58]  B T Thomas Yeo,et al.  Disruption of cortical association networks in schizophrenia and psychotic bipolar disorder. , 2014, JAMA psychiatry.

[59]  Nora D. Volkow,et al.  Temporal Evolution of Brain Functional Connectivity Metrics: Could 7 Min of Rest be Enough? , 2016, Cerebral cortex.

[60]  M. Chun,et al.  Functional connectome fingerprinting: Identifying individuals based on patterns of brain connectivity , 2015, Nature Neuroscience.

[61]  D. Cicchetti,et al.  Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. , 1981, American journal of mental deficiency.

[62]  Thomas E. Nichols,et al.  Best practices in data analysis and sharing in neuroimaging using MRI , 2017, Nature Neuroscience.

[63]  Iluminada Corripio,et al.  Resting-state functional connectivity alterations in the default network of schizophrenia patients with persistent auditory verbal hallucinations , 2015, Schizophrenia Research.

[64]  Edward G. Carmines,et al.  Reliability and Validity Assessment , 1979 .

[65]  Noreen M. Webb,et al.  Generalizability Theory: Overview , 2005 .

[66]  Essa Yacoub,et al.  The WU-Minn Human Connectome Project: An overview , 2013, NeuroImage.

[67]  C. Keown,et al.  Patterns of Atypical Functional Connectivity and Behavioral Links in Autism Differ Between Default, Salience, and Executive Networks. , 2016, Cerebral cortex.

[68]  Elena Deych,et al.  Gibbs distribution for statistical analysis of graphical data with a sample application to fcMRI brain images , 2016, Statistics in medicine.

[69]  R. Shavelson,et al.  Sampling Variability of Performance Assessments. , 1993 .

[70]  Alexandre R. Franco,et al.  Evaluating the reliability of different preprocessing steps to estimate graph theoretical measures in resting state fMRI data , 2015, Front. Neurosci..

[71]  Lars Kai Hansen,et al.  Optimizing the fMRI data-processing pipeline using prediction and reproducibility performance metrics: I. A preliminary group analysis , 2004, NeuroImage.

[72]  Stephen M Smith,et al.  Fast robust automated brain extraction , 2002, Human brain mapping.

[73]  Dustin Scheinost,et al.  Unified Framework for Development, Deployment and Robust Testing of Neuroimaging Algorithms , 2011, Neuroinformatics.

[74]  H. Laufs,et al.  Decoding Wakefulness Levels from Typical fMRI Resting-State Data Reveals Reliable Drifts between Wakefulness and Sleep , 2014, Neuron.

[75]  Aysenil Belger,et al.  Reliability of an fMRI paradigm for emotional processing in a multisite longitudinal study , 2015, Human brain mapping.

[76]  Mary E. Meyerand,et al.  The effect of scan length on the reliability of resting-state fMRI connectivity estimates , 2013, NeuroImage.

[77]  Yufeng Zang,et al.  Toward reliable characterization of functional homogeneity in the human brain: Preprocessing, scan duration, imaging resolution and computational space , 2013, NeuroImage.

[78]  Theo G. M. van Erp,et al.  Reliability of functional magnetic resonance imaging activation during working memory in a multi-site study: Analysis from the North American Prodrome Longitudinal Study , 2014, NeuroImage.