The Unreliability of Measures of Intercoder Reliability , and What to do About it

In both automated and traditional text analysis, human coders are regularly tasked with categorizing documents. Researchers then evaluate the success of this crucial step in the research process via one of many measures of intercoder reliability, such as Cronbachs alpha. They then improve coding practices until this measure reaches some arbitrary threshold, at which point remaining disagreements are resolved in arbitrary ways and ignored in subsequent analyses. We show that this common practice can generate severely biased estimates and misleading conclusions. The problem is the focus on measures of intercoder reliability which, except at the extreme, are unrelated to the quantities of interest, such as the proportion of documents in each category. We thus develop an approach that enables scholars to directly incorporate coding uncertainty into statistical estimation. The method offers an interval estimate which we prove contains the true proportion of documents in each category, under reasonable assumptions. We then extend this method to situations with multiple coders, when one coder is trusted more than another, and when the resulting document codes are used as inputs to another statistical model. We offer easy-to-use software that implements all our suggestions. ∗Department of Political Science, Stanford University, JustinGrimmer.org, jgrimmer@stanford.edu. †Institute for Quantitative Social Science, Harvard University; GaryKing.org, king@harvard.edu, (617) 500-7570. ‡Institute for Quantitative Social Science, Harvard University, scholar.harvard.edu/csuperti, csuperti@fas.harvard.edu 1