General observer-agreement measures on individual subjects and groups of subjects

General chance-corrected measures of agreement on individual subjects, for several observers using nominal or ordinal categories, are developed. The subject-specific measures can be used to identify subjects whom the observers find difficult to rate. The relationship of the subject-specific measures to a general chance-corrected measure of agreement for a group of subjects is demonstrated. By suitable choices of disagreement functions, the measure of agreement for a group of subjects is shown to include, as special cases, many of the kappa-like statistics. Also, it is asymptotically equivalent to various intraclass correlation coefficients. The measures do not require that the observers all use the classification scale in the same way. The asymptotic null and non-null variances obtained by Taylorseries approximations for the statistics are presented. The application of the measures is illustrated by data obtained when seven pathologists classified slides on a five-point ordinal scale for the diagnosis of carcinoma in situ of the uterine cervix.