Randomization tests: A new gold standard?

Classical statistical methods rely on the analytical power of mathematics and some assumptions rather than on computer power. In research with human participants the assumption of random sampling is rarely correct. The great increase in computer power in recent decades makes available an approach to statistical inference which does not require random sampling, namely randomization tests. For these tests we do need random assignment of conditions or treatments to participants or observation occasions, but this is usually necessary anyway to ensure internal validity. External validity is achieved by replication and nonstatistical reasoning whether we use classical or randomization tests. Small-n and single case investigations (including phase designs) benefit from randomization designs and tests, but they can equally well be used for large-n studies. Software is becoming available for analysis of randomization tests, course materials are being developed, and we may be about to see them become a very common if not the principal statistical technique in our toolbox.

[1]  R. Fisher.  Statistical methods for research workers , 1927, Protoplasma.

[2]  William F. Rosenberger,et al.  Sequential monitoring with conditional randomization tests , 2012 .

[3]  Eugene S. Edgington,et al.  Randomization Tests , 2011, International Encyclopedia of Statistical Science.

[4]  B. Manly.  Randomization, Bootstrap and Monte Carlo Methods in Biology , 2018 .

[5]  Karl Pearson F.R.S..  X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling , 2009 .

[6]  B. J. Winer.  Statistical Principles in Experimental Design , 1992 .

[7]  D. C. Howell.  Statistical Methods for Psychology , 1987 .

[8]  Fernando Zúñiga,et al.  Randomization tests in language typology , 2006 .

[9]  Benjamin N. Nelson.  Statistical Methods in Chemistry , 1960 .

[10]  W. F. Scott,et al.  New Cambridge Statistical Tables , 1995 .

[11]  Patrick Onghena,et al.  An R package for single-case randomization tests , 2008, Behavior research methods.

[12]  K. Pearson.  On the Criterion that a Given System of Deviations from the Probable in the Case of a Correlated System of Variables is Such that it Can be Reasonably Supposed to have Arisen from Random Sampling , 1900 .

[13]  B. Efron,et al.  The Jackknife: The Bootstrap and Other Resampling Plans. , 1983 .

[14]  Todd Swanson,et al.  Development and Assessment of a Preliminary Randomization-Based Introductory Statistics Curriculum , 2011 .

[15]  Student,et al.  THE PROBABLE ERROR OF A MEAN , 1908 .

[16]  Robert S. Erikson,et al.  Randomization Tests and Multi-Level Data in U.S. State Politics , 2010, State Politics & Policy Quarterly.

[17]  Bruce E. Trumbo,et al.  Using R to Simulate Permutation Distributions for Some Elementary Experimental Designs , 2010 .

[18]  John Ferron,et al.  Statistical Power of Randomization Tests Used with Multiple-Baseline Designs , 2002 .

[19]  Cedric A. B. Smith,et al.  An Introduction to Genetic Statistics , 1958 .

[20]  D. Stuart-Fox,et al.  Accelerated speciation in colour-polymorphic birds , 2012, Nature.

[21]  J. V. Bradley.  Distribution-Free Statistical Tests , 1968 .

[22]  M. Friedman.  The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance , 1937 .

[23]  E. Pitman.  Significance Tests Which May be Applied to Samples from Any Populations , 1937 .

[24]  William G. Cochran,et al.  Experimental Designs, 2nd Edition , 1950 .

[25]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[26]  John Todman,et al.  Single-case and Small-n Experimental Designs : A Practical Guide To Randomization Tests , 2001 .

[27]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[28]  Matyas and Greenwood.  Serial Dependency in Single-Case Time Series , 2014 .

[29]  Michael R. Chernick,et al.  Resampling Methods , 2002 .

[30]  S. Martin,et al.  The use of randomization tests to assess the degree of similarity in PFGE patterns of E. coli O157 isolates from known outbreaks and statistical space–time clusters , 2006, Epidemiology and Infection.