Data analytic methods for the analysis of multilevel questions: A comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling

Researchers investigating organizations and leadership in particular are increasingly being called upon to theorize multilevel models and to utilize multilevel data analytic techniques. However, the literature provides relatively little guidance for researchers to identify which of the multilevel methodologies are appropriate for their particular questions. In this final article, the statistical procedures used in the multilevel data analyses in the previous articles of this special issue are compared. Specifically, intraclass correlation coefficients (ICCs), rwg(j), hierarchical linear modeling (HLM), within- and between-analysis (WABA), and random group resampling (RGR) are examined and their results and conclusions discussed. Following comparisons of these methods, recommendations for their use are presented.

[1]  Judith D. Singer,et al.  Using SAS PROC MIXED to Fit Multilevel Models, Hierarchical Models, and Individual Growth Models , 1998 .

[2]  C. Schriesheim,et al.  Investigating contingencies: an examination of the impact of span of supervision and upward controllingness on leader-member exchange using traditional and multivariate within- and between-entities analysis. , 2000, The Journal of applied psychology.

[3]  Jennifer M. George,et al.  Personality, affect, and behavior in groups. , 1990 .

[4]  Paul D. Bliese,et al.  Multilevel random coefficient modeling in organizational research: Examples using SAS and S-PLUS. , 2002 .

[5]  Francis J. Yammarino,et al.  On the application of within and between analysis: Are absence and affect really group-based phenomena? , 1992 .

[6]  L. James,et al.  Estimating within-group interrater reliability with and without response bias. , 1984 .

[7]  L. James.  Aggregation Bias in Estimates of Perceptual Agreement. , 1982 .

[8]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[9]  M. Lindell,et al.  Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes. , 2000, The Journal of applied psychology.

[10]  D. Rousseau.  Issues of level in organizational research: Multi-level and cross-level perspectives. , 1985 .

[11]  Lawrence R. James,et al.  Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. , 1993 .

[12]  Chester A. Schriesheim,et al.  Multivariate and moderated within- and between-entity analysis (WABA) using hierarchical linear multiple regression , 1995 .

[13]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .

[14]  William J. Haga,et al.  A vertical dyad linkage approach to leadership within formal organizations: A longitudinal investigation of the role making process , 1975 .

[15]  Francis J. Yammarino,et al.  Multivariate aspects of the varient/waba approach: A discussion and leadership illustration , 1998 .

[16]  Paul D. Bliese,et al.  Group Consensus and Psychological Well‐Being: A Large Field Study1 , 1998 .

[17]  M. Lindell,et al.  Assessing interrater agreement on the job relevance of a test: A comparison of CVI, T, r-sub(WG(J)), and r*-sub(WG(J)) indexes. , 1999 .

[18]  B. Bass.  LEADERSHIP AND PERFORMANCE BEYOND EXPECTATIONS , 1985 .

[19]  David A. Hofmann.  An Overview of the Logic and Rationale of Hierarchical Linear Models , 1997 .

[20]  P. Bliese.  Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. , 2000 .

[21]  P. Bliese,et al.  Group Size and Measures of Group-Level Properties: An Examination of Eta-Squared and ICC Values , 1998 .

[22]  G. Graen,et al.  Relationship-based approach to leadership: Development of leader-member exchange (LMX) theory of leadership over 25 years: Applying a multi-level multi-domain perspective , 1995 .

[23]  L. James,et al.  rwg: An assessment of within-group interrater agreement. , 1993 .

[24]  K. Klein,et al.  Levels Issues in Theory Development, Data Collection, and Analysis , 1994 .

[25]  John M. Charnes,et al.  Estimation of Quantiles for the Sampling Distribution of the rWG within Group Agreement Index , 1995 .

[26]  C. Schriesheim,et al.  Leader–member exchange (LMX) research: A comprehensive review of theory, measurement, and data-analytic practices. , 1999 .

[27]  D. A. Kenny,et al.  Consequences of violating the independence assumption in analysis of variance. , 1986 .

[28]  Paul D. Bliese,et al.  Individual and Nomothetic Models of Job Stress: An Examination of Work Hours, Cohesion, and Well‐Being1 , 1996 .

[29]  Chester A. Schriesheim,et al.  Is it “trustworthy”? A multiple-levels-of-analysis reexamination of an ohio state leadership study, with implications for future research , 1995 .

[30]  Lawrence R. James,et al.  The unmeasured variables problem in path analysis. , 1980 .

[31]  S. Kozlowski,et al.  From Micro to Meso: Critical Steps in Conceptualizing and Conducting Multilevel Research , 2000 .

[32]  P. Bliese.  Group Size, ICC Values, and Group-Level Correlations: A Simulation , 1998 .

[33]  M. Lindell,et al.  A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target , 1999 .

[34]  Lawrence R. James,et al.  Organizational climate: Another look at a potentially important construct. , 1982 .

[35]  Karlene H. Roberts,et al.  Hypothesized Interdependence, Assumed Independence , 1984 .

[36]  Robert C. Blattberg,et al.  Discussant’s Comments , 1979, The Quest for a Science of Accounting.

[37]  S. Kozlowski,et al.  A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. , 1992 .