The Exact Variance of Weighted Kappa with Multiple Raters

Weighted kappa described by Cohen in 1968 is widely used in psychological research to measure agreement between two independent raters. Everitt then provided the exact variance for weighted kappa for two raters. In this paper, Everitt's exact variance is extended to three or more raters.

[1]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[2]  H. Kundel,et al.  Measurement of observer agreement. , 2003, Radiology.

[3]  W. Willett,et al.  Misinterpretation and misuse of the kappa statistic. , 1987, American journal of epidemiology.

[4]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[5]  A Martõ Ân,et al.  Delta: A new measure of agreement , 2022 .

[6]  Brian Everitt,et al.  MOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA , 1968 .

[7]  R. Zwick,et al.  Another look at interrater agreement. , 1988, Psychological bulletin.

[8]  Dale J. Prediger,et al.  Coefficient Kappa: Some Uses, Misuses, and Alternatives , 1981 .

[9]  Christof Schuster,et al.  A Note on the Interpretation of Weighted Kappa and its Relations to Other Rater Agreement Statistics for Metric Scales , 2004 .

[10]  L. Hsu,et al.  Interrater Agreement Measures: Comments on Kappan, Cohen's Kappa, Scott's π, and Aickin's α , 2003 .

[11]  J. D. Mast.  Agreement and Kappa-Type Indices , 2007 .

[12]  P. Mielke,et al.  Cumulant methods for analysing independence of r-way contingency tables and goodness-of-fit frequency data , 1988 .

[13]  C. Schuster,et al.  Dispersion-weighted kappa: An integrative framework for metric and nominal scale agreement coefficients , 2005 .

[14]  A. M. Andrés,et al.  Chance-corrected measures of reliability and validity in K K tables , 2005, Statistical methods in medical research.

[15]  M. Banerjee,et al.  Beyond kappa: A review of interrater agreement measures , 1999 .

[16]  Janis E. Johnston,et al.  A Fortran Program for Computing the Exact Variance of Weighted Kappa , 2005, Perceptual and motor skills.

[17]  J C Nelson,et al.  Statistical description of interrater variability in ordinal ratings , 2000, Statistical methods in medical research.

[18]  A. M. Andrés,et al.  Delta: a new measure of agreement between two raters. , 2004, The British journal of mathematical and statistical psychology.

[19]  P. Graham,et al.  The analysis of ordinal agreement data: beyond weighted kappa. , 1993, Journal of clinical epidemiology.

[20]  B. Everitt,et al.  Large sample standard errors of kappa and weighted kappa. , 1969 .