Reliability of graph analysis of resting state fMRI using test-retest dataset from the Human Connectome Project

The exploration of brain networks with resting-state fMRI (rs-fMRI) combined with graph theoretical approaches has become popular, with the perspective of finding network graph metrics as biomarkers in the context of clinical studies. A preliminary requirement for such findings is to assess the reliability of the graph based connectivity metrics. In previous test-retest (TRT) studies, this reliability has been explored using intraclass correlation coefficient (ICC) with heterogeneous results. But the issue of sample size has not been addressed. Using the large TRT rs-fMRI dataset from the Human Connectome Project (HCP), we computed ICCs and their corresponding p-values (applying permutation and bootstrap techniques) and varied the number of subjects (from 20 to 100), the scan duration (from 400 to 1200 time points), the cost and the graph metrics, using the Anatomic-Automatic Labelling (AAL) parcellation scheme. We quantified the reliability of the graph metrics computed both at global and regional level depending, at optimal cost, on two key parameters, the sample size and the number of time points or scan duration. In the cost range between 20% to 35%, most of the global graph metrics are reliable with 40 subjects or more with long scan duration (14min 24s). In large samples (for instance, 100 subjects), most global and regional graph metrics are reliable for a minimum scan duration of 7min 14s. Finally, for 40 subjects and long scan duration (14min 24s), the reliable regions are located in the main areas of the default mode network (DMN), the motor and the visual networks.

[1]  N. Metropolis,et al.  The Monte Carlo method. , 1949 .

[2]  Hang Joon Jo,et al.  Trouble at Rest: How Correlation Patterns and Group Differences Become Distorted After Global Signal Regression , 2012, Brain Connect..

[3]  Archana Venkataraman,et al.  Intrinsic functional connectivity as a tool for human connectomics: theory, properties, and optimization. , 2010, Journal of neurophysiology.

[4]  E. Haacke,et al.  Imaging iron stores in the brain using magnetic resonance imaging. , 2005, Magnetic resonance imaging.

[5]  Jeremy D. Schmahmann,et al.  Functional topography in the human cerebellum: A meta-analysis of neuroimaging studies , 2009, NeuroImage.

[6]  Mark W. Woolrich,et al.  Resting-state fMRI in the Human Connectome Project , 2013, NeuroImage.

[7]  O. Sporns,et al.  Complex brain networks: graph theoretical analysis of structural and functional systems , 2009, Nature Reviews Neuroscience.

[8]  Habib Benali,et al.  Combining Spatial Independent Component Analysis with Regression to Identify the Subcortical Components of Resting-State fMRI Functional Networks , 2014, Brain Connect..

[9]  N. Volkow,et al.  Laterality patterns of brain functional connectivity: gender effects. , 2012, Cerebral cortex.

[10]  M K Markey,et al.  The reliability of measuring physical characteristics of spiculated masses on mammography. , 2006, The British journal of radiology.

[11]  Olaf Sporns,et al.  Complex network measures of brain connectivity: Uses and interpretations , 2010, NeuroImage.

[12]  R. Müller,et al.  A critical discussion of intraclass correlation coefficients. , 1994, Statistics in medicine.

[13]  Joaquín Goñi,et al.  Changes in structural and functional connectivity among resting-state networks across the human lifespan , 2014, NeuroImage.

[14]  Stephen M. Smith,et al.  Multiplexed Echo Planar Imaging for Sub-Second Whole Brain FMRI and Fast Diffusion Imaging , 2010, PloS one.

[15]  A. Donner,et al.  A comparison of confidence interval methods for the intraclass correlation coefficient. , 1986, Biometrics.

[16]  B. Biswal,et al.  Functional connectivity in the motor cortex of resting human brain using echo‐planar mri , 1995, Magnetic resonance in medicine.

[17]  Yufeng Zang,et al.  Functional brain hubs and their test–retest reliability: A multiband resting-state functional MRI study , 2013, NeuroImage.

[18]  X. Zuo,et al.  Test-retest reliabilities of resting-state FMRI measurements in human brain functional connectomics: A systems neuroscience perspective , 2014, Neuroscience & Biobehavioral Reviews.

[19]  A. Beckett,et al.  AKUFO AND IBARAPA. , 1965, Lancet.

[20]  V. Latora,et al.  Complex networks: Structure and dynamics , 2006 .

[21]  Mary E. Meyerand,et al.  Age-Related Differences in Test-Retest Reliability in Resting-State Brain Functional Connectivity , 2012, PloS one.

[22]  Efstathios D. Gennatas,et al.  Linked Sex Differences in Cognition and Functional Connectivity in Youth. , 2015, Cerebral cortex.

[23]  Dardo Tomasi,et al.  High-Resolution Functional Connectivity Density: Hub Locations, Sensitivity, Specificity, Reproducibility, and Reliability. , 2016, Cerebral cortex.

[24]  Oliver Grimm,et al.  Test–retest reliability of fMRI-based graph theoretical properties during working memory, emotion processing, and resting state , 2014, NeuroImage.

[25]  J M Bland,et al.  Statistical methods for assessing agreement between two methods of clinical measurement , 1986 .

[26]  N. Tzourio-Mazoyer,et al.  Automated Anatomical Labeling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single-Subject Brain , 2002, NeuroImage.

[27]  Edward T. Bullmore,et al.  The discovery of population differences in network community structure: New methods and applications to brain functional networks in schizophrenia , 2012, NeuroImage.

[28]  N Ramnani,et al.  A probabilistic MR atlas of the human cerebellum , 2009, NeuroImage.

[29]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[30]  Jean-Baptiste Poline,et al.  Analysis of a large fMRI cohort: Statistical and methodological issues for group analyses , 2007, NeuroImage.

[31]  Shinichi Nakagawa,et al.  Repeatability for Gaussian and non‐Gaussian data: a practical guide for biologists , 2010, Biological reviews of the Cambridge Philosophical Society.

[32]  Yong He,et al.  Effects of Different Correlation Metrics and Preprocessing Factors on Small-World Brain Functional Networks: A Resting-State Functional MRI Study , 2012, PloS one.

[33]  Kevin Murphy,et al.  The impact of global signal regression on resting state correlations: Are anti-correlated networks introduced? , 2009, NeuroImage.

[34]  E. Bullmore,et al.  Hubs of brain functional networks are radically reorganized in comatose patients , 2012, Proceedings of the National Academy of Sciences.

[35]  T. Boardman.  CONFIDENCE INTERVALS FOR VARIANCE COMPONENTS-A COMPARATIVE MONTE CARLO STUDY , 1974 .

[36]  P. Strick,et al.  Basal Ganglia Output and Cognition: Evidence from Anatomical, Behavioral, and Clinical Studies , 2000, Brain and Cognition.

[37]  H. Duvernoy.  The Human Brain Stem and Cerebellum: Surface, Structure, Vascularization, and Three-Dimensional Sectional Anatomy, with MRI , 2013 .

[38]  Yong He,et al.  Graph Theoretical Analysis of Functional Brain Networks: Test-Retest Evaluation on Short- and Long-Term Resting-State Functional MRI Data , 2011, PloS one.

[39]  A. Donner.  A Review of Inference Procedures for the Intraclass Correlation Coefficient in the One-Way Random Effects Model , 1986 .

[40]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[41]  Mark Jenkinson,et al.  The minimal preprocessing pipelines for the Human Connectome Project , 2013, NeuroImage.

[42]  Yu Wang,et al.  Test–Retest Reliability of Graph Metrics in High‐resolution Functional Connectomics: A Resting‐State Functional MRI Study , 2015, CNS neuroscience & therapeutics.

[43]  Bernard Ng,et al.  Optimization of rs-fMRI Pre-processing for Enhanced Signal-Noise Separation, Test-Retest Reliability, and Group Discrimination , 2015, NeuroImage.

[44]  Simon B. Eickhoff,et al.  One-year test–retest reliability of intrinsic connectivity network fMRI in older adults , 2012, NeuroImage.

[45]  J. Maldjian,et al.  Effect of resting-state functional MR imaging duration on stability of graph theory metrics of brain network connectivity. , 2011, Radiology.

[46]  Anders M. Dale,et al.  An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest , 2006, NeuroImage.

[47]  M. Fox,et al.  Spontaneous fluctuations in brain activity observed with functional magnetic resonance imaging , 2007, Nature Reviews Neuroscience.

[48]  Kevin K Dobbin,et al.  Comparison of confidence interval methods for an intra-class correlation coefficient (ICC) , 2014, BMC Medical Research Methodology.

[49]  Axel Gandy.  Sequential Implementation of Monte Carlo Tests With Uniformly Bounded Resampling Risk , 2009 .

[50]  P. Matthews,et al.  Distinct patterns of brain activity in young carriers of the APOE e4 allele , 2009, NeuroImage.

[51]  Dorothee Auer,et al.  Reproducibility of Graph-Theoretic Brain Network Metrics: A Systematic Review , 2014, Brain Connect..

[52]  Vince D. Calhoun,et al.  Lateralization of resting state networks and relationship to age and gender , 2015, NeuroImage.

[53]  Alan C. Evans,et al.  Three-Dimensional MRI Atlas of the Human Cerebellum in Proportional Stereotaxic Space , 1999, NeuroImage.

[54]  Adam J. Schwarz,et al.  Negative edges and soft thresholding in complex network analysis of resting state functional connectivity data , 2011, NeuroImage.

[55]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .

[56]  Hernando Ombao,et al.  Quantifying temporal correlations: A test–retest evaluation of functional connectivity in resting-state fMRI , 2013, NeuroImage.

[57]  Edward T. Bullmore,et al.  Efficiency and Cost of Economical Brain Functional Networks , 2007, PLoS Comput. Biol..

[58]  John P. A. Ioannidis,et al.  How to Make More Published Research True , 2014, PLoS medicine.

[59]  Antonio Napolitano,et al.  Test-retest reliability of graph metrics of resting state MRI functional brain networks: A review , 2015, Journal of Neuroscience Methods.

[60]  Richard S. Frackowiak,et al.  Improved segmentation of deep brain grey matter structures using magnetization transfer (MT) parameter maps , 2009, NeuroImage.

[61]  Edward T. Bullmore,et al.  Whole-brain anatomical networks: Does the choice of nodes matter? , 2010, NeuroImage.

[62]  Rory A. Fisher,et al.  Theory of Statistical Estimation , 1925, Mathematical Proceedings of the Cambridge Philosophical Society.

[63]  E. Bullmore,et al.  A Resilient, Low-Frequency, Small-World Human Brain Functional Network with Highly Connected Association Cortical Hubs , 2006, The Journal of Neuroscience.

[64]  R Cameron Craddock,et al.  A whole brain fMRI atlas generated via spatially constrained spectral clustering , 2012, Human brain mapping.

[65]  H. Johansen-Berg,et al.  Distinct and overlapping functional zones in the cerebellum defined by resting state functional connectivity. , 2010, Cerebral cortex.

[66]  Axel Gandy,et al.  MMCTest—A Safe Algorithm for Implementing Multiple Monte Carlo Tests , 2012, 1209.3963.

[67]  Andreas Heinz,et al.  Test–retest reliability of resting-state connectivity network characteristics using fMRI and graph theoretical measures , 2012, NeuroImage.

[68]  Jeremy D. Schmahmann,et al.  Functional topography of the cerebellum for motor and cognitive tasks: An fMRI study , 2012, NeuroImage.

[69]  Jonas Richiardi,et al.  Graph analysis of functional brain networks: practical issues in translational neuroscience , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[70]  Mary E. Meyerand,et al.  The effect of scan length on the reliability of resting-state fMRI connectivity estimates , 2013, NeuroImage.

[71]  Jessica A. Grahn,et al.  The role of the basal ganglia in learning and memory: Neuropsychological studies , 2009, Behavioural Brain Research.

[72]  Dongrong Xu,et al.  Altered topological properties of the cortical motor‐related network in patients with subcortical stroke revealed by graph theoretical analysis , 2014, Human brain mapping.

[73]  Edward T. Bullmore,et al.  SYSTEMS NEUROSCIENCE Original Research Article , 2009 .

[74]  D. Cicchetti.  Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. , 1994 .