Permutation inference for the general linear model

Permutation methods can provide exact control of false positives and allow the use of non-standard statistics, making only weak assumptions about the data. With the availability of fast and inexpensive computing, their main limitation would be some lack of flexibility to work with arbitrary experimental designs. In this paper we report on results on approximate permutation methods that are more flexible with respect to the experimental design and nuisance variables, and conduct detailed simulations to identify the best method for settings that are typical for imaging research scenarios. We present a generic framework for permutation inference for complex general linear models (glms) when the errors are exchangeable and/or have a symmetric distribution, and show that, even in the presence of nuisance effects, these permutation inferences are powerful while providing excellent control of false positives in a wide range of common and relevant imaging research scenarios. We also demonstrate how the inference on glm parameters, originally intended for independent data, can be used in certain special but useful cases in which independence is violated. Detailed examples of common neuroimaging applications are provided, as well as a complete algorithm – the “randomise” algorithm – for permutation inference with the glm.

[1]  B. L. Welch ON THE COMPARISON OF SEVERAL MEAN VALUES: AN ALTERNATIVE APPROACH , 1951 .

[2]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[3]  Stephen M. Smith,et al.  Threshold-free cluster enhancement: Addressing problems of smoothing, threshold dependence and localisation in cluster inference , 2009, NeuroImage.

[4]  B. Manly Randomization, Bootstrap and Monte Carlo Methods in Biology , 2018 .

[5]  I. Guttman Linear models : an introduction , 1982 .

[6]  Thomas E. Nichols,et al.  Nonstationary cluster-size inference with random field and permutation methods , 2004, NeuroImage.

[7]  A. W. Kemp,et al.  Randomization, Bootstrap and Monte Carlo Methods in Biology , 1997 .

[8]  Michael Breakspear,et al.  Spatiotemporal wavelet resampling for functional neuroimaging data , 2004, Human brain mapping.

[9]  E. B. Wilson Probable Inference, the Law of Succession, and Statistical Inference , 1927 .

[10]  Y. Benjamini,et al.  Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics , 1999 .

[11]  M. D. Ernst Permutation Methods: A Basis for Exact Inference , 2004 .

[12]  Seuck Heun Song,et al.  A new random permutation test in ANOVA models , 2007 .

[13]  Thomas E. Nichols,et al.  Nonparametric permutation tests for functional neuroimaging: A primer with examples , 2002, Human brain mapping.

[14]  D. B. Duncan,et al.  Estimating Heteroscedastic Variances in Linear Models , 1975 .

[15]  Mert R. Sabuncu,et al.  Measuring and comparing brain cortical surface area and other areal quantities , 2012, NeuroImage.

[16]  E. Bullmore,et al.  Statistical methods of estimation and inference for functional MR image analysis , 1996, Magnetic resonance in medicine.

[17]  Thomas E. Nichols,et al.  Adjusting the effect of nonstationarity in cluster-based and TFCE inference , 2011, NeuroImage.

[18]  A. P. White,et al.  The approximate randomization test as an alternative to the F test in analysis of variance , 1981 .

[19]  Susan R. Wilson,et al.  Two guidelines for bootstrap hypothesis testing , 1991 .

[20]  J D Watson,et al.  Nonparametric Analysis of Statistic Images from Functional Mapping Experiments , 1996, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[21]  E. Pitman SIGNIFICANCE TESTS WHICH MAY BE APPLIED TO SAMPLES FROM ANY POPULATIONS III. THE ANALYSIS OF VARIANCE TEST , 1938 .

[22]  Stephen M. Smith,et al.  General multilevel linear modeling for group analysis in FMRI , 2003, NeuroImage.

[23]  G. Smyth,et al.  Statistical Applications in Genetics and Molecular Biology Permutation P -values Should Never Be Zero: Calculating Exact P -values When Permutations Are Randomly Drawn , 2011 .

[24]  Matthew K. Belmonte,et al.  Permutation testing made practical for functional magnetic resonance image analysis , 2001, IEEE Transactions on Medical Imaging.

[25]  Donald Fraser,et al.  Randomization Tests for a Multivariate Two-Sample Problem , 1958 .

[26]  Brian S. Cade,et al.  PERMUTATION TESTS FOR LEAST ABSOLUTE DEVIATION REGRESSION , 1996 .

[27]  B. Efron Computers and the Theory of Statistics: Thinking the Unthinkable , 1979 .

[28]  MARTI J. ANDERSONa,et al.  PERMUTATION TESTS FOR MULTIFACTORIAL ANALYSIS OF VARIANCE , 2008 .

[29]  Mark W. Woolrich,et al.  Meaningful design and contrast estimability in FMRI , 2007, NeuroImage.

[30]  P. Sen Estimates of the Regression Coefficient Based on Kendall's Tau , 1968 .

[31]  William H. Press,et al.  Numerical recipes in C , 2002 .

[32]  Thomas E. Nichols,et al.  Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate , 2002, NeuroImage.

[33]  William J. Welch,et al.  Construction of Permutation Tests , 1990 .

[34]  F. Pesarin Multivariate Permutation Tests : With Applications in Biostatistics , 2001 .

[35]  David M. Stoneman,et al.  Testing for the Inclusion of Variables in Einear Regression by a Randomisation Technique , 1966 .

[36]  Charles Stein,et al.  On the Theory of Some Non-Parametric Hypotheses , 1949 .

[37]  N C Andreasen,et al.  Tests for Comparing Images Based on Randomization and Permutation Methods , 1996, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[38]  G. Jogesh Babu,et al.  Multivariate Permutation Tests , 2002, Technometrics.

[39]  Eugene S. Edgington,et al.  Randomization Tests , 2011, International Encyclopedia of Statistical Science.

[40]  E. Brunner,et al.  The Nonparametric Behrens‐Fisher Problem: Asymptotic Theory and a Small‐Sample Approximation , 2000 .

[41]  Gerard R. Ridgway,et al.  Statistical analysis for longitudinal MR imaging of dementia , 2009 .

[42]  David Thomas,et al.  The Art in Computer Programming , 2001 .

[43]  Bryan F. J. Manly,et al.  Randomization and regression methods for testing for associations with geographical, environmental and biological distances between populations , 1986, Researches on Population Ecology.

[44]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[45]  David Krackhardt,et al.  Sensitivity of MRQAP Tests to Collinearity and Autocorrelation Conditions , 2007, Psychometrika.

[46]  G. S. James THE COMPARISON OF SEVERAL GROUPS OF OBSERVATIONS WHEN THE RATIOS OF THE POPULATION VARIANCES ARE UNKNOWN , 1951 .

[47]  James F Troendle,et al.  Multiple Testing with Minimal Assumptions , 2008, Biometrical journal. Biometrische Zeitschrift.

[48]  B. M. Brown,et al.  Permutation Tests for Complex Data: Theory, Applications and Software by F. Pesarin and L. Salmaso , 2012 .

[49]  Phillip I. Good,et al.  Extensions Of The Concept Of Exchangeability And Their Applications , 2002 .

[50]  Richard M. Leahy,et al.  A comparison of random field theory and permutation methods for the statistical analysis of MEG data , 2005, NeuroImage.

[51]  R. Fisher THE FIDUCIAL ARGUMENT IN STATISTICAL INFERENCE , 1935 .

[52]  Oscar Kempthorne,et al.  THE RANDOMIZATION THEORY OF' EXPERIMENTAL INFERENCE* , 1955 .

[53]  T A Carpenter,et al.  Colored noise and computational inference in neurophysiological (fMRI) time series analysis: Resampling methods in time and wavelet domains , 2001, Human brain mapping.

[54]  Marti J. Anderson,et al.  Permutation tests for multi-factorial analysis of variance , 2003 .

[55]  E. Edgington Approximate Randomization Tests , 1969 .

[56]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[57]  Felix Famoye,et al.  Plane Answers to Complex Questions: Theory of Linear Models , 2003, Technometrics.

[58]  John Suckling,et al.  Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain , 1999, IEEE Transactions on Medical Imaging.

[59]  S. L. Andersen,et al.  Permutation Theory in the Derivation of Robust Criteria and the Study of Departures from Assumption , 1955 .

[60]  J. J. Higgins,et al.  A Study of Multivariate Permutation Tests Which May Replace Hotelling's T2 Test in Prescribed Circumstances. , 1994, Multivariate behavioral research.

[61]  Peter E. Kennedy Randomization Tests in Econometrics , 1995 .

[62]  P. J. Jennings,et al.  Time series analysis in the time domain and resampling methods for studies of functional magnetic resonance brain imaging , 1997, Human brain mapping.

[63]  Thomas W. O'Gorman,et al.  The Performance of Randomization Tests that Use Permutations of Independent Variables , 2005 .

[64]  M. H. Gail,et al.  Tests for no treatment e?ect in randomized clinical trials , 1988 .

[65]  J. Jastrow,et al.  On small differences in sensation , 1884 .

[66]  M. Dwass Modified Randomization Tests for Nonparametric Hypotheses , 1957 .

[67]  Pierre Legendre,et al.  An empirical comparison of permutation methods for tests of partial regression coefficients in a linear model , 1999 .

[68]  E. Bullmore,et al.  Permutation tests for factorially designed neuroimaging experiments , 2004, Human brain mapping.

[69]  Henry Scheffe,et al.  Statistical Inference in the Non-Parametric Case , 1943 .

[70]  E. J. G. Pitman,et al.  Significance Tests Which May be Applied to Samples from Any Populations. II. The Correlation Coefficient Test , 1937 .

[71]  Marti J. Anderson,et al.  Permutation Tests for Linear Models , 2001 .

[72]  John Ludbrook,et al.  Why Permutation Tests are Superior to t and F Tests in Biomedical Research , 1998 .

[73]  E. S. Pearson SOME ASPECTS OF THE PROBLEM OF RANDOMIZATION , 1937 .

[74]  B. M. Brown,et al.  Distribution‐Free Methods in Regression1 , 1982 .

[75]  Salvador Ruiz-Correa,et al.  Morphology-based hypothesis testing in discrete random fields: A non-parametric method to address the multiple-comparison problem in neuroimaging , 2011, NeuroImage.

[76]  P. Good Permutation, Parametric, and Bootstrap Tests of Hypotheses , 2005 .

[77]  David A. Freedman,et al.  A Nonstochastic Interpretation of Reported Significance Levels , 1983 .

[78]  Bruce Levin,et al.  Urn Models for Regression Analysis, With Applications to Employment Discrimination Studies , 1983 .

[79]  Hannu Oja On Permutation Tests in Multiple Regression and Analysis of Covariance Problems , 1987 .

[80]  Thomas E. Nichols,et al.  Rank-order versus mean based statistics for neuroimaging , 2007, NeuroImage.

[81]  S C Williams,et al.  Generic brain activation mapping in functional magnetic resonance imaging: a nonparametric approach. , 1997, Magnetic resonance imaging.

[82]  Angela R Laird,et al.  Comparison of Fourier and wavelet resampling methods , 2004, Magnetic resonance in medicine.

[83]  C. Braak,et al.  Permutation Versus Bootstrap Significance Tests in Multiple Regression and Anova , 1992 .

[84]  Sara Kherad-Pajouh,et al.  An exact permutation method for testing any effect in balanced and unbalanced fixed effect ANOVA , 2010, Comput. Stat. Data Anal..

[85]  Peter E. Kennedy,et al.  Randomization tests for multiple regression , 1996 .

[86]  A. Aspin Tables for use in comparisons whose accuracy involves two variances, separately estimated. , 1949, Biometrika.

[87]  S. S. Young,et al.  Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment , 1993 .

[88]  Karl-Heinz Jöckel,et al.  Bootstrapping and Related Techniques , 1992 .

[89]  Myoungshic Jhun,et al.  RANDOM PERMUTATION TESTING IN MULTIPLE LINEAR REGRESSION , 2001 .

[90]  E. Pitman Significance Tests Which May be Applied to Samples from Any Populations , 1937 .

[91]  L. Salmaso,et al.  Permutation tests for complex data : theory, applications and software , 2010 .

[92]  A. I.,et al.  Neural Field Continuum Limits and the Structure–Function Partitioning of Cognitive–Emotional Brain Networks , 2023, Biology.

[93]  D. Sengupta Linear models , 2003 .

[94]  Stephen M. Smith,et al.  GLM permutation - nonparametric inference for arbitrary general linear models , 2008 .

[95]  Bryan F. J. Manly,et al.  ANALYSIS OF VARIANCE BY RANDOMIZATION WITH SMALL DATA SETS , 1998 .