Same Data - Different Software - Different Results? Analytic Variability of Group fMRI Results

A plethora of tools and techniques are now available to process and model fMRI data. However, this ‘methodological plurality’ has come with a drawback. Application of different analysis pipelines (Carp, 2012), alterations in software version (Glatard, 2015), and even changes in operating system (Gronenschild, 2012) have all been shown to cause variation in the results of a neuroimaging study. This high analytic flexibility has been pinpointed as a key factor that can lead to increased false-positives (Ioannidis, 2005), and compounded with a lack of data sharing, irreproducible research findings (Poldrack, 2017). In this work, we seek to understand how choice of software package impacts analysis results. We reproduce the results of three published neuroimaging studies (Schonberg, 2012; Moran, 2012; Padmanabhan, 2011) with publicly available data within the three main neuroimaging software packages: AFNI, FSL and SPM, using parametric and nonparametric inference. All information for how to process, analyze, and model each dataset we obtain from the publication. We make a variety of comparisons to assess the similarity of our results across both software packages and choice of inference method.