Small Group Statistics: A Monte Carlo Comparison of Parametric and Randomization Tests

Under some conditions, behavioral research includes statistical tests to assess the probability of experimental outcomes. Occasionally, sample sizes are insufficient to fulfill the assumptions of traditional parametric procedures. The following study assessed the statistical advantages, correspondence, and accuracy of univariate analyses when small-n samples are calculated in both randomized and traditional/parametric formats. Specifically, we compared probability values for both parametric and randomized two-sample independent t-tests across a series of Monte Carlo, pseudorandom two-group data sets of equal size. In doing so, we identified some patterns of probability change associated with decreasing group size.

[1]  Joseph P. Romano.  Bootstrap and randomization tests of some nonparametric hypotheses , 1989 .

[2]  B. Manly.  Randomization, Bootstrap and Monte Carlo Methods in Biology , 2018 .

[3]  B. J. Winer.  Statistical Principles in Experimental Design , 1992 .

[4]  S. Hayes,et al.  The Impact of Acceptance Versus Control Rationales on Pain Tolerance , 1999 .

[5]  Ronald Aylmer Sir Fisher,et al.  141: "The Coefficient of Racial Likeness" and the Future of Craniometry , 1936 .

[6]  G. Glass,et al.  Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analyses of Variance and Covariance , 1972 .

[7]  Tim Gerrodette,et al.  The Uses of Statistical Power in Conservation Biology: The Vaquita and Northern Spotted Owl , 1993 .

[8]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[9]  E. Pitman.  Significance Tests Which May be Applied to Samples from Any Populations , 1937 .

[10]  G. Mcculler,et al.  School and Behavioral Psychology , 2000, Springer US.

[11]  Julian D. Olden,et al.  Assessing the robustness of randomization tests: examples from behavioural studies , 2001, Animal Behaviour.

[12]  Leonard J. Savage,et al.  On Rereading R. A. Fisher , 1976 .

[13]  J. Roscoe.  Fundamental Research Statistics for the Behavioral Sciences , 2004 .

[14]  Prior Learning and Equivalence Class Formation , 1998 .

[15]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[16]  John Todman,et al.  Single-case and Small-n Experimental Designs : A Practical Guide To Randomization Tests , 2001 .

[17]  P. Good,et al.  Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses , 1995 .

[18]  G. Glass,et al.  Basic Statistics for the Behavioral Sciences , 1977 .