Analysis Options for Testing Group Differences on Ordered Categorical Variables: An Empirical Investigation of Type I Error Control and Statistical Power

Type I error control and statistical power of four methods of testing group differences on an ordered categorical response variable were evaluated in a Monte Carlo study. Data were analyzed using the independent means t-test, the chi-square test of homogeneity, the delta statistic, and a cumulative logit model. The number of categories of the response variable, sample size, population distribution shape, and effect size were examined. These experimental conditions were crossed with each other providing a total of 192 conditions. The independent means t-test provided the best control of Type I error, but was rarely the most powerful. For the 5-point response scale, the chi-square was most often the most powerful. Results varied for the 7-point response scale. Small power differences (in many instances) among these procedures suggest that researchers’ choices should be driven by the interpretations that are appropriate for the research questions being addressed.

[1]  N. Cliff.  Ordinal methods for behavioral data analysis , 1996 .

[2]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[3]  J. J. Higgins,et al.  A Comparison of the Power of Wilcoxon's Rank-Sum Statistic to that of Student'st Statistic Under Various Nonnormal Distributions , 1980 .

[4]  P. McCullagh,et al.  Generalized Linear Models, 2nd Edn. , 1990 .

[5]  Mark L. Davison,et al.  Parametric statistics and levels of measurement. , 1988 .

[6]  A. Agresti,et al.  Statistical Methods for the Social Sciences , 1979 .

[7]  A Agresti,et al.  Tutorial on modeling ordered categorical response data. , 1989, Psychological bulletin.

[8]  N. Cliff.  Answering Ordinal Questions with Ordinal Data Using Ordinal Statistics. , 1996, Multivariate behavioral research.

[9]  A. Agresti.  An introduction to categorical data analysis , 1997 .

[10]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[11]  Leonard R. Sussman,et al.  Nominal, Ordinal, Interval, and Ratio Typologies are Misleading , 1993 .

[12]  John A. Nelder,et al.  Generalized linear models. 2nd ed. , 1993 .

[13]  N. Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions. , 1993 .

[14]  J. J. Higgins,et al.  Comparison of the power of the paired samples t test to that of Wilcoxon's signed-ranks test under various population shapes. , 1985 .

[15]  S. Sawilowsky,et al.  Analysis of Likert scale data in disability and medical rehabilitation research. , 1998 .

[16]  A. Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[17]  S. S. Stevens.  Mathematics, measurement, and psychophysics. , 1951 .

[18]  Robert S. Barcikowski,et al.  The Number of Iterations in Monte Carlo Studies of Robustness. , 1988 .