Best Practices in Data Analysis and Sharing in Neuroimaging using MRI

Neuroimaging enables rich noninvasive measurements of human brain activity, but translating such data into neuroscientific insights and clinical applications requires complex analyses and collaboration among a diverse array of researchers. The open science movement is reshaping scientific culture and addressing the challenges of transparency and reproducibility of research. To advance open science in neuroimaging the Organization for Human Brain Mapping created the Committee on Best Practice in Data Analysis and Sharing (COBIDAS), charged with creating a report that collects best practice recommendations from experts and the entire brain imaging community. The purpose of this work is to elaborate the principles of open and reproducible research for neuroimaging using Magnetic Resonance Imaging (MRI), and then distill these principles to specific research practices. Many elements of a study are so varied that practice cannot be prescribed, but for these areas we detail the information that must be reported to fully understand and potentially replicate a study. For other elements of a study, like statistical modelling where specific poor practices can be identified, and the emerging areas of data sharing and reproducibility, we detail both good practice and reporting standards. For each of seven areas of a study we provide tabular listing of over 100 items to help plan, execute, report and share research in the most transparent fashion. Whether for individual scientists, or for editors and reviewers, we hope these guidelines serve as a benchmark, to raise the standards of practice and reporting in neuroimaging using MRI.

[1]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[2]  R. Poldrack Region of interest analysis for fMRI. , 2007, Social cognitive and affective neuroscience.

[3]  Arthur W. Toga,et al.  Provenance in neuroimaging , 2008, NeuroImage.

[4]  Arthur W Toga,et al.  The LONI Pipeline Processing Environment , 2003, NeuroImage.

[5]  Gaël Varoquaux,et al.  Small-sample brain mapping: sparse recovery on spatially correlated designs with randomization and clustering , 2012, ICML.

[6]  Ben Inglis,et al.  A checklist for fMRI acquisition methods reporting in the literature , 2015 .

[7]  Yong He,et al.  Altered baseline brain activity in children with ADHD revealed by resting-state functional MRI. , 2007, Brain & development.

[8]  Chaozhe Zhu,et al.  An improved approach to detection of amplitude of low-frequency fluctuation (ALFF) for resting-state fMRI: Fractional ALFF , 2008, Journal of Neuroscience Methods.

[9]  Christophe Phillips,et al.  Mapping track density changes in nigrostriatal and extranigral pathways in Parkinson's disease , 2014, NeuroImage.

[10]  A M Dale,et al.  Optimal experimental design for event‐related fMRI , 1999, Human brain mapping.

[11]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[12]  Chris Rorden,et al.  Lin4Neuro: a customized Linux distribution ready for neuroimaging analysis , 2011, BMC Medical Imaging.

[13]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[14]  Tristan Glatard,et al.  Reproducibility of neuroimaging analyses across operating systems , 2015, Front. Neuroinform..

[15]  Carole A. Goble,et al.  Towards open science: the myExperiment approach , 2010, Concurr. Comput. Pract. Exp..

[16]  Resource Identification Initiative Members The Resource Identification Initiative: A cultural shift in publishing , 2015 .

[17]  D. Margulies,et al.  Regional Variation in Interhemispheric Coordination of Intrinsic Hemodynamic Fluctuations , 2008, The Journal of Neuroscience.

[18]  Iveta Simera,et al.  EQUATOR: reporting guidelines for health research , 2008, Open medicine : a peer-reviewed, independent, open-access journal.

[19]  L. HARKing: Hypothesizing After the Results are Known , 2002 .

[20]  Arthur W. Toga,et al.  Effi cient , distributed and interactive neuroimaging data analysis using the LONI Pipeline , 2009 .

[21]  Michael L. Waskom,et al.  Frontoparietal Representations of Task Context Support the Flexible Control of Goal-Directed Cognition , 2014, The Journal of Neuroscience.

[22]  Saul Perlmutter,et al.  Blind analysis: Hide results to seek the truth , 2015, Nature.

[23]  Mark W. Woolrich,et al.  The danger of systematic bias in group-level FMRI-lag-based causality estimation , 2012, NeuroImage.

[24]  C. Jack,et al.  Ways toward an early diagnosis in Alzheimer’s disease: The Alzheimer’s Disease Neuroimaging Initiative (ADNI) , 2005, Alzheimer's & Dementia.

[25]  D. Ruppert The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[26]  Karim Jerbi,et al.  Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy , 2015, Journal of Neuroscience Methods.

[27]  Brian A. Nosek,et al.  Registered Reports A Method to Increase the Credibility of Published Results , 2014 .

[28]  Joshua Carp,et al.  The secret lives of experiments: Methods reporting in the fMRI literature , 2012, NeuroImage.

[29]  O. Bagasra,et al.  Proceedings of the National Academy of Sciences , 1914, Science.

[30]  Alan C. Evans,et al.  Multi-level bootstrap analysis of stable clusters in resting-state fMRI , 2009, NeuroImage.

[31]  Andrea Bergmann,et al.  Statistical Parametric Mapping The Analysis Of Functional Brain Images , 2016 .

[32]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[33]  J. Wicherts,et al.  Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results , 2011, PloS one.

[34]  Friedrich Leisch,et al.  Sweave: Dynamic Generation of Statistical Reports Using Literate Data Analysis , 2002, COMPSTAT.

[35]  Ron Mengelers,et al.  The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements , 2012, PloS one.

[36]  Daniel S. Marcus,et al.  The extensible neuroimaging archive toolkit , 2007, Neuroinformatics.

[37]  Russell A. Poldrack,et al.  Guidelines for reporting an fMRI study , 2008, NeuroImage.

[38]  Steven E. Petersen,et al.  The mixed block/event-related design , 2012, NeuroImage.

[39]  Yaroslav O. Halchenko,et al.  Open is Not Enough. Let's Take the Next Step: An Integrated, Community-Driven Computing Platform for Neuroscience , 2012, Front. Neuroinform..

[40]  Abraham Z. Snyder,et al.  Human Connectome Project informatics: Quality control, database services, and data visualization , 2013, NeuroImage.

[41]  Thomas E. Nichols,et al.  Power calculation for group fMRI studies accounting for arbitrary design and temporal autocorrelation , 2008, NeuroImage.

[42]  E. Schlesinger,et al.  The science of scientific writing , 1979, Proceedings of the IEEE.

[43]  Mark Jenkinson,et al.  The minimal preprocessing pipelines for the Human Connectome Project , 2013, NeuroImage.

[44]  Regina Nuzzo,et al.  Scientific method: Statistical errors , 2014, Nature.

[45]  N. Kerr HARKing: Hypothesizing After the Results are Known , 1998, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[46]  Satrajit S. Ghosh,et al.  Data sharing in neuroimaging research , 2012, Front. Neuroinform..

[47]  C. Tufanaru,et al.  Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence , 2015, BMJ : British Medical Journal.

[48]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[49]  David C. Van Essen,et al.  Application of Information Technology: An Integrated Software Suite for Surface-based Analyses of Cerebral Cortex , 2001, J. Am. Medical Informatics Assoc..

[50]  David B. Keator,et al.  Towards structured sharing of raw and derived neuroimaging data across existing resources , 2012, NeuroImage.

[51]  Rhodri Cusack,et al.  Automatic analysis (aa): efficient neuroimaging workflows and parallel processing using Matlab and XML , 2015, Front. Neuroinform..

[52]  Krzysztof J. Gorgolewski,et al.  Making big data open: data sharing in neuroimaging , 2014, Nature Neuroscience.

[53]  G. Zaharchuk,et al.  Recommended implementation of arterial spin-labeled perfusion MRI for clinical applications: A consensus of the ISMRM perfusion study group and the European consortium for ASL in dementia. , 2015, Magnetic resonance in medicine.

[54]  Alan C. Evans,et al.  The pipeline system for Octave and Matlab (PSOM): a lightweight scripting framework and execution engine for scientific workflows , 2012, Front. Neuroinform..

[55]  Yaroslav O. Halchenko,et al.  Four aspects to make science open “by design” and not as an after-thought , 2015, GigaScience.

[56]  E. Kanal,et al.  ACR guidance document on MR safe practices: 2013 , 2013, Journal of magnetic resonance imaging : JMRI.

[57]  Satrajit S. Ghosh,et al.  Nipype: A Flexible, Lightweight and Extensible Neuroimaging Data Processing Framework in Python , 2011, Front. Neuroinform..

[58]  H. Benali,et al.  BrainVISA: Software platform for visualization and analysis of multi-modality brain data , 2001, NeuroImage.

[59]  Thomas E. Nichols,et al.  Post-hoc power estimation for topological inference in fMRI , 2014, NeuroImage.

[60]  Tristan Glatard,et al.  The MNI data-sharing and processing ecosystem , 2016, NeuroImage.

[61]  R. Buxton,et al.  Detection Power, Estimation Efficiency, and Predictability in Event-Related fMRI , 2001, NeuroImage.

[62]  C. Drummond Replicability is not Reproducibility:Nor is it Good Science , 2009 .

[63]  Thomas R. Knösche,et al.  White matter integrity, fiber count, and other fallacies: The do's and don'ts of diffusion MRI , 2013, NeuroImage.

[64]  Bruce R. Rosen,et al.  Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measures , 2015, Scientific Data.

[65]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[66]  Marcia McNutt,et al.  Journals unite for reproducibility , 2014, Science.

[67]  Olga V. Demler,et al.  Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. , 2005, Archives of general psychiatry.

[68]  Sebastián M. Real,et al.  E2F1 Regulates Cellular Growth by mTORC1 Signaling , 2011, PloS one.

[69]  Matthew H. Brush,et al.  The Resource Identification Initiative: A cultural shift in publishing , 2015, F1000Research.

[70]  Thomas E. Nichols,et al.  Diagnosis and exploration of massively univariate neuroimaging models , 2003, NeuroImage.

[71]  J. Mumford A power calculation guide for fMRI studies. , 2012, Social cognitive and affective neuroscience.

[72]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[73]  Erik Schultes,et al.  The FAIR Guiding Principles for scientific data management and stewardship , 2016, Scientific Data.

[74]  Jessica A. Turner,et al.  COINS: An Innovative Informatics and Neuroimaging Tool Suite Built for Large Heterogeneous Datasets , 2011, Front. Neuroinform..

[75]  J. Ioannidis Why Most Published Research Findings Are False , 2019, CHANCE.

[76]  Arno Klein,et al.  Large-scale evaluation of ANTs and FreeSurfer cortical thickness measurements , 2014, NeuroImage.

[77]  Y. Zang,et al.  Altered baseline brain activity in children with ADHD revealed by resting-state functional MRI , 2007, Brain and Development.

[78]  Hans Knutsson,et al.  Empirically investigating the statistical validity of SPM, FSL and AFNI for single subject fMRI analysis , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[79]  R. Peng Reproducible Research in Computational Science , 2011, Science.

[80]  Sean Bechhofer,et al.  Towards open science: the myExperiment approach , 2010 .

[81]  Yingli Lu,et al.  Regional homogeneity approach to fMRI data analysis , 2004, NeuroImage.

[82]  Andreas Daffertshofer,et al.  Comparing Brain Networks of Different Size and Connectivity Density Using Graph Theory , 2010, PloS one.

[83]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[84]  Published Online Biomedical research: increasing value, reducing waste , 2014 .

[85]  P. Habibzadeh,et al.  Decay of References to Web sites in Articles Published in General Medical Journals: Mainstream vs Small Journals , 2013, Applied Clinical Informatics.

[86]  Yu-Kang Tu,et al.  Simpson's Paradox, Lord's Paradox, and Suppression Effects are the same phenomenon – the reversal paradox , 2008, Emerging themes in epidemiology.

[87]  Lars Kai Hansen,et al.  The Quantitative Evaluation of Functional Neuroimaging Experiments: The NPAIRS Data Analysis Framework , 2000, NeuroImage.

[88]  L. Christophorou Science , 2018, Emerging Dynamics: Science, Energy, Society and Values.

[89]  Satoru Hayasaka,et al.  Development of PowerMap: a Software Package for Statistical Power Calculation in Neuroimaging Studies , 2012, Neuroinformatics.

[90]  H. Pashler,et al.  Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition 1 , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[91]  Alan C. Evans,et al.  LORIS: a web-based data management system for multi-center studies , 2012, Front. Neuroinform..

[92]  I. Cuthill,et al.  Reporting : The ARRIVE Guidelines for Reporting Animal Research , 2010 .

[93]  Gilles Faÿ,et al.  Características inmunológicas claves en la fisiopatología de la sepsis. Infectio , 2009 .

[94]  U. Dirnagl,et al.  Biomedical research: increasing value, reducing waste , 2014, The Lancet.

[95]  David W Jordan,et al.  PET/MR Imaging Consensus Paper: A Joint Paper by the Society of Nuclear Medicine and Molecular Imaging Technologist Section and the Section for Magnetic Resonance Technologists , 2013, The Journal of Nuclear Medicine Technology.

[96]  Thomas E. Nichols,et al.  Optimization of experimental design in fMRI: a general framework using a genetic algorithm , 2003, NeuroImage.