Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes

Within the context of climate strength, this simulation study examines the validity of various dispersion indexes for detecting meaningful relationships between variability in group member perceptions and outcome variables. We used the simulation to model both individual-and group-level phenomena, vary appropriate population characteristics, and test the proclivity of standard and average deviation, interrater agreement indexes (rwg, r*wg, awg), and coefficient of variation (both normed and unnormed) for Type I and Type II errors. The results show that the coefficient of variation was less likely to detect interaction effects although it outperformed other measures when detecting level effects. Standard deviation was shown to be inferior to other indexes when no level effect is present although it may be an effective measure of dispersion when modeling strength or interaction effects. The implications for future research, in which dispersion is a critical component of the theoretical model, are discussed.

[1]  Michael K. Lindell,et al.  Recent Issues Regarding rWG, rWG, rWG(J), and rWG(J) , 2005 .

[2]  Chester A. Schriesheim,et al.  Is it “trustworthy”? A multiple-levels-of-analysis reexamination of an ohio state leadership study, with implications for future research , 1995 .

[3]  E Doveh,et al.  Statistical properties of the rWG(J) index of agreement. , 2001, Psychological methods.

[4]  Neil M. A. Hauenstein,et al.  Interrater Agreement Reconsidered: An Alternative to the rwg Indices , 2005 .

[5]  S. Kozlowski,et al.  A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. , 1992 .

[6]  Arthur G. Bedeian,et al.  On the Use of the Coefficient of Variation as a Measure of Diversity , 2000 .

[7]  D. Chan.  Functional Relations among Constructs in the Same Content Domain at Different Levels of Analysis: A Typology of Composition Models , 1998 .

[8]  Michael J. Burke,et al.  Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide , 2002 .

[9]  M. Lindell,et al.  A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target , 1999 .

[10]  D. Hofmann,et al.  The Structure and Function of Collective Constructs: Implications for Multilevel Research and Theory Development , 1999 .

[11]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[12]  J. Peiró,et al.  An examination of the antecedents and moderator influences of climate strength. , 2002, The Journal of applied psychology.

[13]  Arnon E. Reichers,et al.  On the Etiology of Climates. , 1983 .

[14]  Quinetta M. Roberson,et al.  Justice in Teams: A Review of Fairness Effects in Collective Contexts , 2005 .

[15]  Michael J. Burke,et al.  On Average Deviation Indices for Estimating Interrater Agreement , 1999 .

[16]  M. Smithson.  On relative dispersion: A new solution for some old problems , 1982 .

[17]  P. Allison.  Measures of Inequality , 1978 .

[18]  L. James,et al.  Estimating within-group interrater reliability with and without response bias. , 1984 .

[19]  S. Stigler.  Studies in the History of Probability and Statistics. XXXII Laplace, Fisher, and the discovery of the concept of sufficiency , 1973 .

[20]  John E. Hunter,et al.  Interrater reliability coefficients cannot be computed when only one stimulus is rated. , 1989 .

[21]  M. Lindell,et al.  Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes. , 2000, The Journal of applied psychology.

[22]  Stephanie L. Castro,et al.  Data analytic methods for the analysis of multilevel questions: A comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling , 2002 .

[23]  Raymond A. Noe,et al.  JUSTICE IN TEAMS: ANTECEDENTS AND CONSEQUENCES OF PROCEDURAL JUSTICE CLIMATE , 2002 .

[24]  B. Schneider,et al.  Climate strength: a new direction for climate research. , 2002, The Journal of applied psychology.

[25]  Ayala Cohen,et al.  Statistical properties of the r-sub(WG(J) ) index of agreement. , 2001 .

[26]  Mark Schaller,et al.  Methods for the Measurement of Consensual Beliefs Within Groups , 1998 .

[27]  P. Bliese.  Group Size, ICC Values, and Group-Level Correlations: A Simulation , 1998 .

[28]  L. James,et al.  rwg: An assessment of within-group interrater agreement. , 1993 .

[29]  S. Kozlowski,et al.  Multilevel Theory, Research, a n d M e t h o d s i n Organizations Foundations, Extensions, and New Directions , 2022 .

[30]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[31]  W. Mischel.  Toward a cognitive social learning reconceptualization of personality. , 1973, Psychological review.

[32]  S. Kozlowski,et al.  From Micro to Meso: Critical Steps in Conceptualizing and Conducting Multilevel Research , 2000 .

[33]  Arthur G. Bedeian,et al.  Cross-Level Inference and Organizational Research: Perspectives on Interpretation and Application , 1983 .

[34]  Louis N. Gray,et al.  Measurement of Relative Variation: Sociological Examples , 1971 .

[35]  P. Bliese.  Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. , 2000 .