Best Practices in Data Analysis and Sharing in Neuroimaging using MRI

Neuroimaging enables rich noninvasive measurements of human brain activity, but translating such data into neuroscientific insights and clinical applications requires complex analyses and collaboration among a diverse array of researchers. The open science movement is reshaping scientific culture and addressing the challenges of transparency and reproducibility of research. To advance open science in neuroimaging the Organization for Human Brain Mapping created the Committee on Best Practice in Data Analysis and Sharing (COBIDAS), charged with creating a report that collects best practice recommendations from experts and the entire brain imaging community. The purpose of this work is to elaborate the principles of open and reproducible research for neuroimaging using Magnetic Resonance Imaging (MRI), and then distill these principles to specific research practices. Many elements of a study are so varied that practice cannot be prescribed, but for these areas we detail the information that must be reported to fully understand and potentially replicate a study. For other elements of a study, like statistical modelling where specific poor practices can be identified, and the emerging areas of data sharing and reproducibility, we detail both good practice and reporting standards. For each of seven areas of a study we provide tabular listing of over 100 items to help plan, execute, report and share research in the most transparent fashion. Whether for individual scientists, or for editors and reviewers, we hope these guidelines serve as a benchmark, to raise the standards of practice and reporting in neuroimaging using MRI.

[1]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[2]  Erik Schultes,et al.  The FAIR Guiding Principles for scientific data management and stewardship , 2016, Scientific Data.

[3]  Tristan Glatard,et al.  The MNI data-sharing and processing ecosystem , 2016, NeuroImage.

[4]  Benedikt V. Ehinger,et al.  Faculty Opinions recommendation of PSYCHOLOGY. Estimating the reproducibility of psychological science. , 2015 .

[5]  Saul Perlmutter,et al.  Blind analysis: Hide results to seek the truth , 2015, Nature.

[6]  C. Tufanaru,et al.  Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence , 2015, BMJ : British Medical Journal.

[7]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[8]  Karim Jerbi,et al.  Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy , 2015, Journal of Neuroscience Methods.

[9]  Yaroslav O. Halchenko,et al.  Four aspects to make science open “by design” and not as an after-thought , 2015, GigaScience.

[10]  Bruce R. Rosen,et al.  Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measures , 2015, Scientific Data.

[11]  David N. Kennedy,et al.  The Resource Identification Initiative: A cultural shift in publishing , 2015, Neuroinformatics.

[12]  Ben Inglis,et al.  A checklist for fMRI acquisition methods reporting in the literature , 2015 .

[13]  Tristan Glatard,et al.  Reproducibility of neuroimaging analyses across operating systems , 2015, Front. Neuroinform..

[14]  Hans Knutsson,et al.  Empirically investigating the statistical validity of SPM, FSL and AFNI for single subject fMRI analysis , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[15]  Rhodri Cusack,et al.  Automatic analysis (aa): efficient neuroimaging workflows and parallel processing using Matlab and XML , 2015, Front. Neuroinform..

[16]  G. Zaharchuk,et al.  Recommended implementation of arterial spin-labeled perfusion MRI for clinical applications: A consensus of the ISMRM perfusion study group and the European consortium for ASL in dementia. , 2015, Magnetic resonance in medicine.

[17]  Marcia McNutt,et al.  Journals unite for reproducibility , 2014, Science.

[18]  Krzysztof J. Gorgolewski,et al.  Making big data open: data sharing in neuroimaging , 2014, Nature Neuroscience.

[19]  Christophe Phillips,et al.  Mapping track density changes in nigrostriatal and extranigral pathways in Parkinson's disease , 2014, NeuroImage.

[20]  Arno Klein,et al.  Large-scale evaluation of ANTs and FreeSurfer cortical thickness measurements , 2014, NeuroImage.

[21]  Michael L. Waskom,et al.  Frontoparietal Representations of Task Context Support the Flexible Control of Goal-Directed Cognition , 2014, The Journal of Neuroscience.

[22]  Brian A. Nosek,et al.  Registered Reports A Method to Increase the Credibility of Published Results , 2014 .

[23]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[24]  Regina Nuzzo,et al.  Scientific method: Statistical errors , 2014, Nature.

[25]  U. Dirnagl,et al.  Biomedical research: increasing value, reducing waste , 2014, The Lancet.

[26]  Thomas E. Nichols,et al.  Post-hoc power estimation for topological inference in fMRI , 2014, NeuroImage.

[27]  Abraham Z. Snyder,et al.  Human Connectome Project informatics: Quality control, database services, and data visualization , 2013, NeuroImage.

[28]  P. Habibzadeh Decay of References to Web sites in Articles Published in General Medical Journals: Mainstream vs Small Journals , 2013, Applied Clinical Informatics.

[29]  Mark Jenkinson,et al.  The minimal preprocessing pipelines for the Human Connectome Project , 2013, NeuroImage.

[30]  David W Jordan,et al.  PET/MR Imaging Consensus Paper: A Joint Paper by the Society of Nuclear Medicine and Molecular Imaging Technologist Section and the Section for Magnetic Resonance Technologists , 2013, The Journal of Nuclear Medicine Technology.

[31]  Thomas R. Knösche,et al.  White matter integrity, fiber count, and other fallacies: The do's and don'ts of diffusion MRI , 2013, NeuroImage.

[32]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[33]  E. Kanal,et al.  ACR guidance document on MR safe practices: 2013 , 2013, Journal of magnetic resonance imaging : JMRI.

[34]  J. Carp The secret lives of experiments: Methods reporting in the fMRI literature , 2012, NeuroImage.

[35]  H. Beek F1000Prime recommendation of False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. , 2012 .

[36]  David B. Keator,et al.  Towards structured sharing of raw and derived neuroimaging data across existing resources , 2012, NeuroImage.

[37]  Steven E. Petersen,et al.  The mixed block/event-related design , 2012, NeuroImage.

[38]  J. Mumford A power calculation guide for fMRI studies. , 2012, Social cognitive and affective neuroscience.

[39]  Yaroslav O. Halchenko,et al.  Open is Not Enough. Let's Take the Next Step: An Integrated, Community-Driven Computing Platform for Neuroscience , 2012, Front. Neuroinform..

[40]  Gaël Varoquaux,et al.  Small-sample brain mapping: sparse recovery on spatially correlated designs with randomization and clustering , 2012, ICML.

[41]  Ron Mengelers,et al.  The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements , 2012, PloS one.

[42]  Satoru Hayasaka,et al.  Development of PowerMap: a Software Package for Statistical Power Calculation in Neuroimaging Studies , 2012, Neuroinformatics.

[43]  Satrajit S. Ghosh,et al.  Data sharing in neuroimaging research , 2012, Front. Neuroinform..

[44]  Alan C. Evans,et al.  The pipeline system for Octave and Matlab (PSOM): a lightweight scripting framework and execution engine for scientific workflows , 2012, Front. Neuroinform..

[45]  Alan C. Evans,et al.  LORIS: a web-based data management system for multi-center studies , 2012, Front. Neuroinform..

[46]  Mark W. Woolrich,et al.  The danger of systematic bias in group-level FMRI-lag-based causality estimation , 2012, NeuroImage.

[47]  R. Peng Reproducible Research in Computational Science , 2011, Science.

[48]  J. Wicherts,et al.  Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results , 2011, PloS one.

[49]  Jessica A. Turner,et al.  COINS: An Innovative Informatics and Neuroimaging Tool Suite Built for Large Heterogeneous Datasets , 2011, Front. Neuroinform..

[50]  Satrajit S. Ghosh,et al.  Nipype: A Flexible, Lightweight and Extensible Neuroimaging Data Processing Framework in Python , 2011, Front. Neuroinform..

[51]  Chris Rorden,et al.  Lin4Neuro: a customized Linux distribution ready for neuroimaging analysis , 2011, BMC Medical Imaging.

[52]  Sebastián M. Real,et al.  E2F1 Regulates Cellular Growth by mTORC1 Signaling , 2011, PloS one.

[53]  Catharine van Ingen,et al.  Towards open science: the myExperiment approach , 2010 .

[54]  C. Stam,et al.  Comparing Brain Networks of Different Size and Connectivity Density Using Graph Theory , 2010, PloS one.

[55]  I. Cuthill,et al.  Reporting : The ARRIVE Guidelines for Reporting Animal Research , 2010 .

[56]  Gilles Faÿ,et al.  Características inmunológicas claves en la fisiopatología de la sepsis. Infectio , 2009 .

[57]  Alan C. Evans,et al.  Multi-level bootstrap analysis of stable clusters in resting-state fMRI , 2009, NeuroImage.

[58]  C. Drummond Replicability is not Reproducibility:Nor is it Good Science , 2009 .

[59]  H. Pashler,et al.  Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition 1 , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[60]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[61]  D. Margulies,et al.  Regional Variation in Interhemispheric Coordination of Intrinsic Hemodynamic Fluctuations , 2008, The Journal of Neuroscience.

[62]  Arthur W. Toga,et al.  Provenance in neuroimaging , 2008, NeuroImage.

[63]  Chaozhe Zhu,et al.  An improved approach to detection of amplitude of low-frequency fluctuation (ALFF) for resting-state fMRI: Fractional ALFF , 2008, Journal of Neuroscience Methods.

[64]  D. Moher,et al.  EQUATOR: reporting guidelines for health research , 2008, The Lancet.

[65]  Thomas E. Nichols,et al.  Guidelines for reporting an fMRI study , 2008, NeuroImage.

[66]  Yu-Kang Tu,et al.  Simpson's Paradox, Lord's Paradox, and Suppression Effects are the same phenomenon – the reversal paradox , 2008, Emerging themes in epidemiology.

[67]  Thomas E. Nichols,et al.  Power calculation for group fMRI studies accounting for arbitrary design and temporal autocorrelation , 2008, NeuroImage.

[68]  Yong He,et al.  Altered baseline brain activity in children with ADHD revealed by resting-state functional MRI. , 2007, Brain & development.

[69]  R. Poldrack Region of interest analysis for fMRI. , 2007, Social cognitive and affective neuroscience.

[70]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[71]  John P. A. Ioannidis,et al.  Why Most Published Research Findings Are False , 2005, PLoS Medicine.

[72]  C. Jack,et al.  Ways toward an early diagnosis in Alzheimer’s disease: The Alzheimer’s Disease Neuroimaging Initiative (ADNI) , 2005, Alzheimer's & Dementia.

[73]  Olga V. Demler,et al.  Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. , 2005, Archives of general psychiatry.

[74]  James Franklin The elements of statistical learning: data mining, inference and prediction , 2005 .

[75]  Yingli Lu,et al.  Regional homogeneity approach to fMRI data analysis , 2004, NeuroImage.

[76]  Thomas E. Nichols,et al.  Diagnosis and exploration of massively univariate neuroimaging models , 2003, NeuroImage.

[77]  Arthur W Toga,et al.  The LONI Pipeline Processing Environment , 2003, NeuroImage.

[78]  Thomas E. Nichols,et al.  Optimization of experimental design in fMRI: a general framework using a genetic algorithm , 2003, NeuroImage.

[79]  H. Benali,et al.  BrainVISA: Software platform for visualization and analysis of multi-modality brain data , 2001, NeuroImage.

[80]  R. Buxton,et al.  Detection Power, Estimation Efficiency, and Predictability in Event-Related fMRI , 2001, NeuroImage.

[81]  Lars Kai Hansen,et al.  The Quantitative Evaluation of Functional Neuroimaging Experiments: The NPAIRS Data Analysis Framework , 2000, NeuroImage.

[82]  N. Kerr HARKing: Hypothesizing After the Results are Known , 1998, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[83]  Andrea Bergmann,et al.  Statistical Parametric Mapping The Analysis Of Functional Brain Images , 2016 .

[84]  Published Online Biomedical research: increasing value, reducing waste , 2014 .

[85]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[86]  Daniel S. Marcus,et al.  The extensible neuroimaging archive toolkit , 2007, Neuroinformatics.

[87]  Friedrich Leisch,et al.  Sweave: Dynamic Generation of Statistical Reports Using Literate Data Analysis , 2002, COMPSTAT.

[88]  L. HARKing: Hypothesizing After the Results are Known , 2002 .

[89]  John W. Harwell,et al.  Application of Information Technology: An Integrated Software Suite for Surface-based Analyses of Cerebral Cortex , 2001, J. Am. Medical Informatics Assoc..

[90]  A. M. Dale Optimal experimental design for event‐related fMRI , 1999, Human brain mapping.

[91]  E. Schlesinger,et al.  The science of scientific writing , 1979, Proceedings of the IEEE.

[92]  John P A Ioannidis Essay Modeling the Framework for False Positive Findings Why Most Published Research Findings Are False , 2022 .

[93]  David B. Keator,et al.  Derived Data Storage and Exchange Workflow for Large-Scale Neuroimaging Analyses on the BIRN Grid , 2009, Front. Neuroinform..