Treating Words as Data with Error: Uncertainty in Text Statements of Policy Positions

Political text offers extraordinary potential as a source of information about the policy positions of political actors. Despite recent advances in computational text analysis, human interpretative coding of text remains an important source of text-based data, ultimately required to validate more automatic techniques. The profession’s main source of cross-national, time-seriesdataonpartypolicypositionscomesfromthehumaninterpretativecodingofpartymanifestosbytheComparative Manifesto Project (CMP). Despite widespread use of these data, the uncertainty associated with each point estimate has neverbeenavailable,underminingthevalueofthedatasetasascientificresource.Weproposearemedy.First,wecharacterize processes by which CMP data are generated. These include inherently stochastic processes of text authorship, as well as of the parsing and coding of observed text by humans. Second, we simulate these error-generating processes by bootstrapping analysesofcodedquasi-sentences.Thisallowsustoestimatepreciselevelsofnonsystematicerrorforeverycategoryandscale reported by the CMP for its entire set of 3,000-plus manifestos. Using our estimates of these errors, we show how to correct biased inferences, in recent prominently published work, derived from statistical analyses of error-contaminated CMP data.