Using Pooled Kappa to Summarize Interrater Agreement across Many Items

We propose the pooled estimator of kappa, an efficient estimator when summarizing the interrater agreement for qualitative data with many items but few subjects. We evaluate this estimator through a simulation of proposed and alternative (average kappa) estimators and subsequently apply our method to calculate pooled and average kappas over 2,176 rated items from six semistructured interviews with sponsors of the CAHPS. The proposed pooled kappa estimator efficiently summarizes interrater agreement by domain. It is more widely applicable and makes better use of scarce subjects than simply averaging item-level kappas.

[1]  H J Schouten.  Estimating kappa from binocular data and comparing marginal probabilities. , 1993, Statistics in medicine.

[2]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[3]  Alexander von Eye.  An Alternative to Cohen's κ , 2006 .

[4]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[5]  Takuji Nishimura,et al.  Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator , 1998, TOMC.

[6]  Alexander Basilevsky,et al.  The ratio estimator and maximum-likelihood weighted least squares regression , 1980 .

[7]  Dale J. Prediger,et al.  Coefficient Kappa: Some Uses, Misuses, and Alternatives , 1981 .

[8]  Patricia Simon,et al.  Including Omission Mistakes in the Calculation of Cohen's Kappa and an Analysis of the Coefficient's Paradox Features , 2006 .

[9]  Joseph P. Newhouse,et al.  Future Directions for the National Health Expenditure Accounts: Conference Overview , 2006, Health care financing review.

[10]  Philip Smith.  Roger Bakeman John M. Gottman , 1987, Animal Behaviour.

[11]  Marc N. Elliott,et al.  Understanding the Reporting Practices of CAHPS® Sponsors , 2007, Health care financing review.

[12]  Roger Bakeman,et al.  The development of symbol-infused joint engagement. , 2004, Child development.

[13]  H. Bernard.  Research Methods in Anthropology: Qualitative and Quantitative Approaches , 1988 .

[14]  Richard L Kravitz,et al.  Characterizing patient requests and physician responses in office practice. , 2002, Health services research.

[15]  J. R. Lockwood,et al.  How important is exact balance in treatment and control sample sizes to evaluations? , 2007, Journal of substance abuse treatment.

[16]  N L Oden.  Estimating kappa from binocular data. , 1991, Statistics in medicine.