Interval Estimates of Multivariate Effect Sizes

Monte Carlo methods were used to examine techniques for constructing confidence intervals around multivariate effect sizes. Using interval inversion and bootstrapping methods, confidence intervals were constructed around the standard estimate of Mahalanobis distance (D 2), two bias-adjusted estimates of D 2, and Huberty’s I. Interval coverage and width were examined across conditions by adjusting sample size, number of variables, population effect size, population distribution shape, and the covariance structure. The accuracy and precision of the intervals varied considerably across methods and conditions; however, the interval inversion approach appears to be promising for D 2, whereas the percentile bootstrap approach is recommended for the other effect size measures. The results imply that it is possible to obtain fairly accurate coverage estimates for multivariate effect sizes. However, interval width estimates tended to be large and uninformative, suggesting that future efforts might focus on investigating design factors that facilitate more precise estimates of multivariate effect sizes.

[1]  G. Cumming,et al.  A Primer on the Understanding, Use, and Calculation of Confidence Intervals that are Based on Central and Noncentral Distributions , 2001 .

[2]  Carl J. Huberty,et al.  Group Overlap as a Basis for Effect Size , 2000 .

[3]  P. Rao,et al.  The jackknife procedure for the probabilities of misclassification , 1985 .

[4]  B. Reiser.  CONFIDENCE INTERVALS FOR THE MAHALANOBIS DISTANCE , 2001 .

[5]  B. Efron,et al.  A Leisurely Look at the Bootstrap, the Jackknife, and , 1983 .

[6]  Bruce Thompson,et al.  Computing Correct Confidence Intervals for Anova Fixed-and Random-Effects Effect Sizes , 2001 .

[7]  W. N. Venables.  Calculation of Confidence Intervals for Noncentrality Parameters , 1975 .

[8]  Ronald C. Serlin,et al.  Rationality in psychological research: The good-enough principle. , 1985 .

[9]  J. Kromrey,et al.  NNCORR: A SAS/IML Program for Generating Nonnormal Correlated Data , 1997 .

[10]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[11]  James H. McMillan,et al.  Reporting Effect Size: The Road Less Traveled. , 2002 .

[12]  J. Jackson Barnette,et al.  The Data Analysis Dilemma: Ban or Abandon. A Review of Null Hypothesis Significance Testing. , 1998 .

[13]  J. Stevens.  Applied Multivariate Statistics for the Social Sciences , 1986 .

[14]  C. J. Huberty,et al.  Applied Discriminant Analysis , 1994 .

[15]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[16]  R. Grissom,et al.  Review of assumptions and problems in the appropriate conceptualization of effect size. , 2001, Psychological methods.

[17]  James P. Stevens,et al.  Power of the multivariate analysis of variance tests. , 1980 .

[18]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[19]  Robert A. Stine,et al.  An Introduction to Bootstrap Methods , 1989 .

[20]  R. Shanmugam.  Multivariate Analysis: Part 2: Classification, Covariance Structures and Repeated Measurements , 1998 .

[21]  James H. Steiger,et al.  R2: A computer program for interval estimation, power Calculations, sample size estimation, and hypothesis testing in multiple regression , 1992 .

[22]  J Carpenter,et al.  Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. , 2000, Statistics in medicine.

[23]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[24]  B. K. Ghosh,et al.  A Comparison of Some Approximate Confidence Intervals for the Binomial Parameter , 1979 .

[25]  Allen I. Fleishman.  A method for simulating non-normal distributions , 1978 .

[26]  Bruce Thompson,et al.  Statistical Significance and Effect Size Reporting: Portrait of a Possible Future. , 1998 .

[27]  Robert S. Barcikowski,et al.  The Number of Iterations in Monte Carlo Studies of Robustness. , 1988 .

[28]  Henry F. Inman,et al.  The overlapping coefficient as a measure of agreement between probability distributions and point estimation of the overlap of two normal densities , 1989 .