Agreement and Information in the Reliability of Coding

Coefficients that assess the reliability of data-making processes, that is, coding text, transcribing interviews, or categorizing observations into analyzable terms, are mostly conceptualized in terms of the agreement a set of coders, observers, judges, or measuring instruments exhibit. When variation is low, reliability coefficients reveal their dependency on an often neglected phenomenon, the amount of information that reliability data provide about the reliability of the coding process or the data it generates. This paper explores the concept of reliability, simple agreement, three conceptions of chance to correct that agreement, and sources of information deficiency, and it develops two measures of information about reliability, akin to the power of a statistical test, intended as a companion to traditional reliability coefficients, especially Krippendorff's (2004, pp. 221–250; Hayes & Krippendorff, 2007) alpha.

[1]  Klaus Krippendorff,et al.  Answering the Call for a Standard Reliability Measure for Coding Data , 2007 .

[2]  Klaus Krippendorff,et al.  Systematic and Random Disagreement and the Reliability of Nominal Data , 2008 .

[3]  A. E. Maxwell.  Comparing the Classification of Subjects by Two Independent Judges , 1970, British Journal of Psychiatry.

[4]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[5]  Dale J. Prediger,et al.  Coefficient Kappa: Some Uses, Misuses, and Alternatives , 1981 .

[6]  Steven D. Brown,et al.  Handbook of applied multivariate statistics and mathematical modeling , 2000 .

[7]  H. Kraemer,et al.  2 x 2 kappa coefficients: measures of agreement or association. , 1989, Biometrics.

[8]  L. Lin,et al.  A concordance correlation coefficient to evaluate reproducibility. , 1989, Biometrics.

[9]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[10]  R. Zwick,et al.  Another look at interrater agreement. , 1988, Psychological bulletin.

[11]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[12]  J. Guilford,et al.  A Note on the G Index of Agreement , 1964 .

[13]  Karl Pearson,et al.  Mathematical contributions to the theory of evolution. IX.—On the principle of homotyposis and its relation to heredity, to the variability of the individual, and to that of the race. Part I.—homotyposis in the vegetable kingdom , 1901, Proceedings of the Royal Society of London.

[14]  W. A. Scott,et al.  Reliability of Content Analysis ; The Case of Nominal Scale Cording , 1955 .

[15]  Karl Pearson,et al.  Mathematical Contributions to the Theory of Evolution. IX. On the Principle of Homotyposis and Its Relation to Heredity, to the Variability of the Individual, and to that of the Race. Part I. Homotypos in the Vegetable Kingdom , 1901 .

[16]  W. D. Perreault,et al.  Reliability of Nominal Data Based on Qualitative Judgments , 1989 .

[17]  R. Alpert,et al.  Communications Through Limited-Response Questioning , 1954 .

[18]  J. Vegelius,et al.  On Generalizations Of The G Index And The Phi Coefficient To Nominal Scales. , 1979, Multivariate behavioral research.

[19]  L. A. Goodman,et al.  Measures of association for cross classifications , 1979 .

[20]  D. Weiss,et al.  Interrater reliability and agreement. , 2000 .

[21]  C. Nickerson.  A note on a concordance correlation coefficient to evaluate reproducibility , 1997 .

[22]  K. Krippendorff.  Quantitative guidelines for communicable disease control programs. , 1978, Biometrics.

[23]  Klaus Krippendorff,et al.  Computing Krippendorff's Alpha-Reliability , 2011 .