MACS – a new SPM toolbox for model assessment, comparison and selection

BACKGROUND In cognitive neuroscience, functional magnetic resonance imaging (fMRI) data are widely analyzed using general linear models (GLMs). However, model quality of GLMs for fMRI is rarely assessed, in part due to the lack of formal measures for statistical model inference. NEW METHOD We introduce a new SPM toolbox for model assessment, comparison and selection (MACS) of GLMs applied to fMRI data. MACS includes classical, information-theoretic and Bayesian methods of model assessment previously applied to GLMs for fMRI as well as recent methodological developments of model selection and model averaging in fMRI data analysis. RESULTS The toolbox - which is freely available from GitHub - directly builds on the Statistical Parametric Mapping (SPM) software package and is easy-to-use, general-purpose, modular, readable and extendable. We validate the toolbox by reproducing model selection and model averaging results from earlier publications. COMPARISON WITH EXISTING METHODS A previous toolbox for model diagnosis in fMRI has been discontinued and other approaches to model comparison between GLMs have not been translated into reusable computational resources in the past. CONCLUSIONS Increased attention on model quality will lead to lower false-positive rates in cognitive neuroscience and increased application of the MACS toolbox will increase the reproducibility of GLM analyses and is likely to increase the replicability of fMRI studies.

[1]  Karl J. Friston,et al.  EEG and MEG Data Analysis in SPM8 , 2011, Comput. Intell. Neurosci..

[2]  Tobias U. Hauser,et al.  The PhysIO Toolbox for Modeling Physiological Noise in fMRI Data , 2017, Journal of Neuroscience Methods.

[3]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[4]  Karl J. Friston,et al.  Dynamic causal modelling , 2003, NeuroImage.

[5]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[6]  H. Jeffreys.  A Treatise on Probability , 1922, Nature.

[7]  Karl J. Friston,et al.  Bayesian fMRI time series analysis with spatial priors , 2005, NeuroImage.

[8]  H. Akaike.  A new look at the statistical model identification , 1974 .

[9]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[10]  Satrajit S. Ghosh,et al.  Nipype: A Flexible, Lightweight and Extensible Neuroimaging Data Processing Framework in Python , 2011, Front. Neuroinform..

[11]  David R. Anderson,et al.  Kullback-Leibler information as a basis for strong inference in ecological studies , 2001 .

[13]  William D. Penny,et al.  Comparing Dynamic Causal Models using AIC, BIC and Free Energy , 2012, NeuroImage.

[14]  Jean-Baptiste Poline,et al.  Improving functional magnetic resonance imaging reproducibility , 2015, GigaScience.

[15]  John-Dylan Haynes,et al.  How to avoid mismodelling in GLM-based fMRI data analysis: cross-validated Bayesian model selection , 2016, NeuroImage.

[16]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[17]  Adrian E. Raftery,et al.  Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors , 1999 .

[18]  William D. Penny,et al.  Efficient Posterior Probability Mapping Using Savage-Dickey Ratios , 2013, PloS one.

[19]  Guillaume Flandin,et al.  Bayesian comparison of spatially regularised general linear models , 2007, Human brain mapping.

[20]  Joshua Carp,et al.  On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments , 2012, Front. Neurosci..

[21]  Yves Rosseel,et al.  On the Definition of Signal-To-Noise Ratio and Contrast-To-Noise Ratio for fMRI Data , 2013, PloS one.

[22]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[23]  Mark W. Woolrich,et al.  Fully Bayesian spatio-temporal modeling of FMRI data , 2004, IEEE Transactions on Medical Imaging.

[24]  Stefan Bode,et al.  Orientation pop-out processing in human visual cortex , 2013, NeuroImage.

[25]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[26]  Karl J. Friston,et al.  Statistical parametric maps in functional imaging: A general linear approach , 1994 .

[27]  Karl J. Friston,et al.  Variational free energy and the Laplace approximation , 2007, NeuroImage.

[28]  Karl J. Friston,et al.  Classical and Bayesian Inference in Neuroimaging: Theory , 2002, NeuroImage.

[29]  Karl J. Friston,et al.  Generalisability, Random Effects & Population Inference , 1998, NeuroImage.

[30]  Karl J. Friston,et al.  Comparing dynamic causal models , 2004, NeuroImage.

[31]  Karl J. Friston,et al.  Comparing Families of Dynamic Causal Models , 2010, PLoS Comput. Biol..

[32]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[33]  Karl J. Friston,et al.  Bayesian model selection for group studies — Revisited , 2014, NeuroImage.

[34]  Karl J. Friston,et al.  Comparing hemodynamic models with DCM , 2007, NeuroImage.

[35]  David Gal,et al.  Abandon Statistical Significance , 2017, The American Statistician.

[36]  John P O'Doherty,et al.  Model-based approaches to neuroimaging: combining reinforcement learning theory with fMRI data. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[37]  Henrik Walter,et al.  Prediction error as a linear function of reward probability is coded in human nucleus accumbens , 2006, NeuroImage.

[38]  Patrick Monahan,et al.  Model assessment and model building in fMRI , 2003, Human brain mapping.

[39]  Martin M. Monti,et al.  Human Neuroscience , 2022 .

[40]  Ju Lynn Ong,et al.  A Bayesian model selection approach to fMRI activation detection , 2010, 2010 IEEE International Conference on Image Processing.

[41]  Jean-Baptiste Poline,et al.  Ambiguous Results in Functional Neuroimaging Data Analysis Due to Covariate Correlation , 1999, NeuroImage.

[42]  Thomas E. Nichols,et al.  Diagnosis and exploration of massively univariate neuroimaging models , 2003, NeuroImage.

[43]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[44]  William D. Penny,et al.  Bayesian model selection maps for group studies , 2009, NeuroImage.

[45]  Karl J. Friston,et al.  Variational Bayesian inference for fMRI time series , 2003, NeuroImage.

[46]  T. Shallice,et al.  Face repetition effects in implicit and explicit memory tests as measured by fMRI. , 2002, Cerebral cortex.