Statistical Significance Levels of Nonparametric Tests Biased by Heterogeneous Variances of Treatment Groups

Abstract The statistical significance levels of the Wilcoxon-Mann-Whitney test and the Kruskal-Wallis test are substantially biased by heterogeneous variances of treatment groups—even when sample sizes are equal. Under these conditions, the Type I error probabilities of the nonparametric tests, performed at the .01, .05, and .10 significance levels, increase by as much as 40%-50% in many cases and sometimes as much as 300%. The bias increases systematically as the ratio of standard deviations of treatment groups increases and remains fairly constant for various sample sizes. There is no indication that Type I error probabilities approach the significance level asymptotically as sample size increases.

[1]  H. Keselman,et al.  Is the ANOVA F-Test Robust to Variance Heterogeneity When Sample Sizes are Equal?: An Investigation via a Coefficient of Variation , 1977 .

[2]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[3]  B. L. Welch.  THE SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO MEANS WHEN THE POPULATION VARIANCES ARE UNEQUAL , 1938 .

[4]  M. E. Muller,et al.  A Note on the Generation of Random Normal Deviates , 1958 .

[5]  H. Scheffé,et al.  The Analysis of Variance , 1960 .

[6]  E. Lehmann,et al.  Nonparametrics: Statistical Methods Based on Ranks , 1976 .

[7]  J E Overall,et al.  Tests That are Robust against Variance Heterogeneity in k × 2 Designs with Unequal Cell Frequencies , 1995, Psychological reports.

[8]  D. W. Zimmerman,et al.  Rank Transformations and the Power of the Student T Test and Welch T' Test for Non-Normal Populations with Unequal Variances , 1993 .

[9]  T. A. Bray,et al.  A Convenient Method for Generating Normal Variables , 1964 .

[10]  Welch Bl.  THE GENERALIZATION OF ‘STUDENT'S’ PROBLEM WHEN SEVERAL DIFFERENT POPULATION VARLANCES ARE INVOLVED , 1947 .

[11]  S. Hora.  Statistical Inference Based on Ranks , 1986 .

[12]  D. W. Zimmerman.  A Note on Homogeneity of Variance of Scores and Ranks , 1996 .

[13]  F. E. Satterthwaite.  An approximate distribution of estimates of variance components. , 1946, Biometrics.

[14]  R. Randles,et al.  Introduction to the Theory of Nonparametric Statistics , 1991 .

[15]  George Marsaglia,et al.  Toward a universal random number generator , 1987 .

[16]  Satterthwaite Fe.  An approximate distribution of estimates of variance components. , 1946 .

[17]  P. Hsu.  Contribution to the theory of "Student's" t-test as applied to the problem of two samples. , 1938 .