Uncertainty over Uncertainty: Investigating the Assumptions, Annotations, and Text Measurements of Economic Policy Uncertainty

Methods and applications are inextricably linked in science, and in particular in the domain of text-as-data. In this paper, we examine one such text-as-data application, an established economic index that measures economic policy uncertainty from keyword occurrences in news. This index, which is shown to correlate with firm investment, employment, and excess market returns, has had substantive impact in both the private sector and academia. Yet, as we revisit and extend the original authors' annotations and text measurements we find interesting text-as-data methodological research questions: (1) Are annotator disagreements a reflection of ambiguity in language? (2) Do alternative text measurements correlate with one another and with measures of external predictive validity? We find for this application (1) some annotator disagreements of economic policy uncertainty can be attributed to ambiguity in language, and (2) switching measurements from keyword-matching to supervised machine learning classifiers results in low correlation, a concerning implication for the validity of the index.

[1]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[2]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[3]  Dacheng Xiu,et al.  The Structure of Economic News , 2020, SSRN Electronic Journal.

[4]  James Pustejovsky,et al.  FactBank: a corpus annotated with event factuality , 2009, Lang. Resour. Evaluation.

[5]  Dhanya Sridhar,et al.  Adapting Text Embeddings for Causal Inference , 2020, UAI.

[6]  Daniel Hernández-Lobato,et al.  Ambiguity Helps: Classification with Disagreements in Crowdsourced Annotations , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Maite Taboada,et al.  Lexicon-Based Methods for Sentiment Analysis , 2011, CL.

[8]  Matt Taddy,et al.  Text As Data , 2017, Journal of Economic Literature.

[9]  Mark Dredze,et al.  Challenges of Using Text Classifiers for Causal Inference , 2018, EMNLP.

[10]  Daniel Jurafsky,et al.  Deconfounded Lexicon Induction for Interpretable Social Science , 2018, NAACL.

[11]  S. Davis,et al.  Measuring Economic Policy Uncertainty , 2013 .

[12]  Hal Daumé,et al.  Deep Unordered Composition Rivals Syntactic Methods for Text Classification , 2015, ACL.

[13]  Paul Ormerod,et al.  Text as data: a machine learning-based approach to measuring uncertainty , 2020, 2006.06457.

[14]  Lora Aroyo,et al.  Crowdsourcing Ground Truth for Medical Relation Extraction , 2017, ACM Trans. Interact. Intell. Syst..

[15]  Brendan T. O'Connor,et al.  Computational Text Analysis for Social Science: Model Assumptions and Complexity , 2011 .

[16]  James A. Evans,et al.  Machine Translation: Mining Text for Social Theory , 2016 .

[17]  Iryna Gurevych,et al.  Cross-Genre and Cross-Domain Detection of Semantic Uncertainty , 2012, CL.

[18]  Jure Leskovec,et al.  Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora , 2016, EMNLP.

[19]  Shimon Kogan,et al.  Information, Trading, and Volatility: Evidence from Firm-Specific News , 2016, The Review of financial studies.

[20]  Arman Cohan,et al.  Longformer: The Long-Document Transformer , 2020, ArXiv.

[21]  Doug Downey,et al.  Abductive Commonsense Reasoning , 2019, ICLR.

[22]  Justin Grimmer,et al.  Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts , 2013, Political Analysis.

[23]  M. Grimaldi,et al.  The Information Content of Central Bank Minutes , 2012 .

[24]  János Csirik,et al.  The CoNLL-2010 Shared Task: Learning to Detect Hedges and their Scope in Natural Language Text , 2010, CoNLL Shared Task.

[25]  Leif Anders Thorsrud.  Words are the New Numbers: A Newsy Coincident Index of the Business Cycle , 2018, Journal of Business & Economic Statistics.

[26]  Andrés Azqueta-Gavaldón,et al.  Developing news-based Economic Policy Uncertainty index with unsupervised machine learning , 2017 .

[27]  Sanjeev Arora,et al.  A Simple but Tough-to-Beat Baseline for Sentence Embeddings , 2017, ICLR.

[28]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[29]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[30]  Delip Rao,et al.  Semi-Supervised Polarity Lexicon Induction , 2009, EACL.

[31]  Dallas Card,et al.  The Importance of Calibration for Estimating Proportions from Annotations , 2018, NAACL.

[32]  Margaret E. Roberts,et al.  Adjusting for Confounding with Text Matching , 2020 .

[33]  Ron Artstein,et al.  Survey Article: Inter-Coder Agreement for Computational Linguistics , 2008, CL.

[34]  Paul C. Tetlock.  Giving Content to Investor Sentiment: The Role of Media in the Stock Market , 2005, The Journal of Finance.

[35]  Michael McMahon,et al.  Transparency and deliberation within the FOMC: a computational linguistics approach , 2014 .

[36]  Huseyin Gulen,et al.  Policy Uncertainty and Corporate Investment , 2015 .

[37]  Francesco Trebbi,et al.  Measuring Central Bank Communication: An Automated Approach with Application to FOMC Statements , 2009 .

[38]  Julieta Yung,et al.  A machine learning approach to identifying different types of uncertainty , 2018, Economics Letters.

[39]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[40]  Dragomir R. Radev,et al.  How to Analyze Political Attention with Minimal Assumptions and Costs , 2010 .

[41]  Jonathan Brogaard,et al.  The Asset-Pricing Implications of Government Economic Policy Uncertainty , 2015, Manag. Sci..

[42]  Ellie Pavlick,et al.  Inherent Disagreements in Human Textual Inferences , 2019, Transactions of the Association for Computational Linguistics.

[43]  Sergei Nirenburg,et al.  Mood and modality: out of theory and into the fray , 2004, Nat. Lang. Eng..

[44]  Gerard Hoberg,et al.  Text-Based Network Industries and Endogenous Product Differentiation , 2010, Journal of Political Economy.

[45]  Illtyd Trethowan.  Causality , 1938 .

[46]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[47]  Katherine A. Keith,et al.  Uncertainty-aware generative models for inferring document class prevalence , 2018, EMNLP.

[48]  Katherine A. Keith,et al.  Modeling Financial Analysts’ Decision Making via the Pragmatics and Semantics of Earnings Calls , 2019, ACL.

[49]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[50]  Udo Kruschwitz,et al.  Comparing Bayesian Models of Annotation , 2018, TACL.

[51]  Margaret E. Roberts,et al.  How to make causal inferences using texts , 2018, Science advances.

[52]  Yejin Choi,et al.  Social IQA: Commonsense Reasoning about Social Interactions , 2019, EMNLP 2019.

[53]  Katherine A. Keith,et al.  Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates , 2020, ACL.

[54]  Yejin Choi,et al.  Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning , 2019, EMNLP.

[55]  Rashmi Prasad,et al.  Annotation and Data Mining of the Penn Discourse TreeBank , 2004, ACL 2004.

[56]  Abigail Z. Jacobs,et al.  Measurement and Fairness , 2019, FAccT.

[57]  J. Loevinger.  Objective Tests as Instruments of Psychological Theory , 1957 .

[58]  Michael Strube,et al.  Finding Hedges by Chasing Weasels: Hedge Detection Using Wikipedia Tags and Shallow Linguistic Features , 2009, ACL.

[59]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[60]  David G. Rand,et al.  Structural Topic Models for Open‐Ended Survey Responses , 2014, American Journal of Political Science.

[61]  悠太 菊池,et al.  大規模要約資源としてのNew York Times Annotated Corpus , 2015 .