rwg: An assessment of within-group interrater agreement.

Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation

[1]  L. Cronbach.  Response Sets and Test Validity , 1946 .

[2]  L. Cronbach.  Further Evidence on Response Sets and Test Design , 1950 .

[3]  B. J. Winer.  Statistical Principles in Experimental Design , 1992 .

[4]  R. H. Finn.  A Note on Estimating the Reliability of Categorical Data , 1970 .

[5]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[6]  G F Lawlis,et al.  Judgment of counseling process: reliability, agreement, and error. , 1972, Psychological bulletin.

[7]  D. Weiss,et al.  Interrater reliability and agreement of subjective judgments , 1975 .

[8]  J. Bartko,et al.  On Various Intraclass Correlation Reliability Coefficients , 1976 .

[9]  John J. Bartko.  Reply to Algina. , 1978 .

[10]  S. K. Mitchell.  Interobserver agreement, reliability, and generalizability of data collected in observational studies. , 1979 .

[11]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[12]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[13]  Fritz Drasgow,et al.  Item response theory : application to psychological measurement , 1983 .

[14]  R. Downey,et al.  Intraclass Correlations: There's More There Than Meets the Eye , 1983 .

[15]  L. James,et al.  Estimating within-group interrater reliability with and without response bias. , 1984 .

[16]  B. Schneider,et al.  Employee and customer perceptions of service in banks: Replication and extension. , 1985 .

[17]  Steve W. J. Kozlowski,et al.  AN EXPLORATION OF CLIMATES FOR TECHNICAL UPDATING AND PERFORMANCE , 1987 .

[18]  B. Bass,et al.  Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership. , 1988 .

[19]  W. W. Stine.  Interobserver relational agreement , 1989 .

[20]  John E. Hunter,et al.  Interrater reliability coefficients cannot be computed when only one stimulus is rated. , 1989 .

[21]  S. Kozlowski,et al.  A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. , 1992 .