Maximum likelihood estimation of agreement in the constant predictive probability model, and its relation to Cohen's kappa.

The alpha agreement parameter is defined as the proportion of a population of items that are classified identically "for cause" by two classifiers, the remaining items being classified at random. The parameters of the corresponding constant predictive probability model are shown to be estimable by the method of maximum likelihood, and a simulation study indicates applicability of the asymptotic results to finite samples. The new estimator tends to be larger than Cohen's kappa, except in the case of uniform margins. An application is made to the validity of cancer risk items included in a cancer registry.

[1]  W. A. Scott,et al.  Reliability of Content Analysis ; The Case of Nominal Scale Cording , 1955 .

[2]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[3]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[4]  B. Everitt,et al.  Large sample standard errors of kappa and weighted kappa. , 1969 .

[5]  R. Light.  Measures of response agreement for qualitative data: Some generalizations and alternatives. , 1971 .

[6]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[7]  J. R. Landis,et al.  A one-way components of variance model for categorical data , 1977 .

[8]  Joseph L. Fleiss,et al.  Comparison of the Null Distributions of Weighted Kappa and the C Ordinal Statistic , 1977 .

[9]  J. Richard Landis,et al.  Large sample variance of kappa in the case of different sets of raters. , 1979 .

[10]  J. Fleiss,et al.  The Reliability of Dichotomous Judgments: Unequal Numbers of Judges per Subject , 1979 .

[11]  A. J. Conger.  Integration and generalization of kappas for multiple raters. , 1980 .

[12]  J. R. Bergan.  MEASURING OBSERVER AGREEMENT USING THE QUASI‐INDEPENDENCE CONCEPT , 1980 .

[13]  A Quasi-Equiprobability Model for Measuring Observer Agreement , 1980 .

[14]  H. Kraemer,et al.  Extension of the kappa coefficient. , 1980, Biometrics.

[15]  Dale J. Prediger,et al.  Coefficient Kappa: Some Uses, Misuses, and Alternatives , 1981 .

[16]  Domenic V. Cicchetti,et al.  Testing the Normal Approximation and Minimal Sample Size Requirements of Weighted Kappa When the Number of Categories is Large , 1981 .

[17]  J. Fleiss,et al.  Measuring Agreement for Multinomial Data , 1982 .

[18]  Annette J. Dobson,et al.  General observer-agreement measures on individual subjects and groups of subjects , 1984 .

[19]  R C Brownson,et al.  A study of the accuracy of cancer risk factor information reported to a central registry compared with that obtained by interview. , 1989, American journal of epidemiology.