On the Marginal Dependency of Cohen’s κ

Cohen’s κ (kappa) is typically used as a measure of degree of rater agreement. It is often criticized because it is marginal-dependent. In this article, this characteristic is explained and illustrated in the context of (1) nonuniform marginal probability distributions, (2) odds ratios that remain constant while κ changes in the presence of varying marginal distributions, and (3) percentages of raw agreement that remain constant while κ changes in the presence of varying marginal distributions. The meaning and interpretation of κ are explained with reference to the log-linear main effect model of variable independence. This model is used for the estimation of the expected cell frequencies of agreement tables. It is shown that the interpretation of κ as a measure of degree of agreement is incorrect. The correct interpretation is that κ assesses the degree of agreement beyond that expected based on a statistical model such as the independence or the null model. Based on Goodman’s (1991) distinction between ...