Multivariate Randomization Tests for Small-N Behavioral Research: A Web-Based Application

Behavioral research involving statistical analyses of small group data is frequently compromised by conventional parametric statistical procedures. As an alternative, we have developed and deployed several web-based applications that allow behavioral researchers to easily input data on-line and to calculate levels of significance for small-n studies. Previously, our web-based applications were restricted to parametric and randomization tests involving only one dependent variable. We now have expanded our algorithms such that multivariate analyses may be conducted on sample sizes as small as 6 while employing several dependent measures. This paper provides details for on-line input and interpretation of randomized multivariate statistical tests for small-n studies. Also, to test the power and reliability of our applications, we have compared our multivariate randomization algorithms against a traditional multivariate statistic with two dependent variables. Using Monte Carlo methods, we have assessed the statistical advantages and accuracy of applied multivariate analyses when calculated in both randomized and traditional/parametric formats. Specifically, we have compared probability values for both traditional and randomized MANOVA models by way of Hotelling’s T2 and the Randomized multivariate/composite z scores. We discuss the reliability problems associated with using traditional multivariate statistics with small-n studies, and we describe the statistical advantages and some limitations of using our on-line, small-n, multivariate randomization tests.

[1]  John Todman,et al.  Single-case and Small-n Experimental Designs : A Practical Guide To Randomization Tests , 2001 .

[2]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[3]  Roger Mundry,et al.  Testing related samples with missing values: a permutation approach , 1999, Animal Behaviour.

[4]  R. Hastings,et al.  Adults' Responses to Self-Injurious Behavior , 1995, Behavior modification.

[5]  G. Glass,et al.  Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analyses of Variance and Covariance , 1972 .

[6]  H. Hotelling.  The Generalization of Student’s Ratio , 1931 .

[7]  Michael L. Davidson.  Univariate versus multivariate tests in repeated-measures experiments. , 1972 .

[8]  J. Roscoe.  Fundamental Research Statistics for the Behavioral Sciences , 2004 .

[9]  Julian D. Olden,et al.  Assessing the robustness of randomization tests: examples from behavioural studies , 2001, Animal Behaviour.

[10]  P. Good,et al.  Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses , 1995 .

[11]  G. Glass,et al.  Basic Statistics for the Behavioral Sciences , 1977 .

[12]  M. Sidman.  Tactics of Scientific Research , 1960 .

[13]  Donald Fraser,et al.  Randomization Tests for a Multivariate Two-Sample Problem , 1958 .

[14]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[15]  David W. Johnson,et al.  Cooperation and Competition: Theory and Research , 1989 .

[16]  J. Gentle,et al.  Randomization and Monte Carlo Methods in Biology. , 1990 .

[17]  S. Kollins,et al.  Experimental analysis of childhood psychopathology: A laboratory matching analysis of the behavior of children diagnosed with Attention-Deficit Hyperactivity Disorder (ADHD) , 1997 .

[18]  E. Pitman.  Significance Tests Which May be Applied to Samples from Any Populations , 1937 .

[19]  R. Poulin,et al.  Using randomization techniques to analyse fluctuating asymmetry data , 1997, Animal Behaviour.

[20]  D. Chiszar,et al.  Rattlesnake orientation to prey as a function of thermal backgrounds and edges , 1997 .

[21]  G. Mcculler,et al.  School and Behavioral Psychology , 2000, Springer US.

[22]  B. Manly.  Randomization, Bootstrap and Monte Carlo Methods in Biology , 2018 .

[23]  J. Stevens.  Applied Multivariate Statistics for the Social Sciences , 1986 .

[24]  Chris Ninness,et al.  Small Group Statistics: A Monte Carlo Comparison of Parametric and Randomization Tests , 2002 .