How We Do Things With Words: Analyzing Text as Social and Cultural Data

In this article we describe our experiences with computational text analysis involving rich social and cultural concepts. We hope to achieve three primary goals. First, we aim to shed light on thorny issues not always at the forefront of discussions about computational text analysis methods. Second, we hope to provide a set of key questions that can guide work in this area. Our guidance is based on our own experiences and is therefore inherently imperfect. Still, given our diversity of disciplinary backgrounds and research practices, we hope to capture a range of ideas and identify commonalities that resonate for many. This leads to our final goal: to help promote interdisciplinary collaborations. Interdisciplinary insights and partnerships are essential for realizing the full potential of any computational text analysis involving social and cultural concepts, and the more we bridge these divides, the more fruitful we believe our work will be.

[1]  Benedikt Szmrecsanyi,et al.  A statistical method for the identification and aggregation of regional linguistic variation , 2011 .

[2]  Chris Welty,et al.  Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard , 2013 .

[3]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001, Statistical Science.

[4]  Corina Koolen,et al.  These are not the Stereotypes You are Looking For: Bias and Fairness in Authorial Gender Attribution , 2017, EthNLP@EACL.

[5]  Daniel Jurafsky,et al.  How to Ask for a Favor: A Case Study on the Success of Altruistic Requests , 2014, ICWSM.

[6]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001 .

[7]  Barbara McGillivray,et al.  Room to Glo: A Systematic Comparison of Semantic Change Detection Approaches with Word Embeddings , 2019, EMNLP.

[8]  Graeme Hirst,et al.  A Tale of Two Cultures: Bringing Literary Analysis and Computational Linguistics Together , 2013, CLfL@NAACL-HLT.

[9]  Shion Guha,et al.  Comparing grounded theory and topic modeling: Extreme divergence or unlikely convergence? , 2017, J. Assoc. Inf. Sci. Technol..

[10]  David Bamman,et al.  A Bayesian Mixed Effects Model of Literary Character , 2014, ACL.

[11]  Cristian Danescu-Niculescu-Mizil,et al.  Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions , 2016, WWW.

[12]  Erik Bleich,et al.  The effect of terrorist events on media portrayals of Islam and Muslims: evidence from New York Times headlines, 1985–2013 , 2016 .

[13]  Kenneth Benoit,et al.  Validating Estimates of Latent Traits from Textual Data Using Human Judgment as a Benchmark , 2012, Political Analysis.

[14]  David Bamman,et al.  Gender identity and lexical variation in social media , 2012, 1210.4567.

[15]  William R. Frey,et al.  Artificial Intelligence and Inclusion: Formerly Gang-Involved Youth as Domain Experts for Analyzing Unstructured Twitter Data , 2018, Social science computer review.

[16]  Cynthia Rudin,et al.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead , 2018, Nature Machine Intelligence.

[17]  Andrew Piper,et al.  Think Small: On Literary Modeling , 2017, PMLA/Publications of the Modern Language Association of America.

[18]  Jennifer L Stevens Aubrey,et al.  Looking Good Versus Feeling Good: An Investigation of Media Frames of Health Advice and Their Effects on Women’s Body-related Self-perceptions , 2010 .

[19]  Erik Bleich,et al.  Media Portrayals of Minorities: Muslims in British Newspaper Headlines, 2001–2012 , 2015 .

[20]  Jacob Eisenstein,et al.  What to do about bad language on the internet , 2013, NAACL.

[21]  Alexander Binder,et al.  Unmasking Clever Hans predictors and assessing what machines really learn , 2019, Nature Communications.

[22]  Matthew J. Salganik,et al.  Bit by bit: social research in the digital age , 2019, The Journal of mathematical sociology.

[23]  Matthew L. Jockers.  Macroanalysis: Digital Methods and Literary History , 2013 .

[24]  Arkaitz Zubiaga,et al.  Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads , 2015, PloS one.

[25]  M. Kirschenbaum.  The Remaking of Reading : Data Mining and the Digital Humanities , 2007 .

[26]  Dong Nguyen,et al.  Why Gender and Age Prediction from Tweets is Hard: Lessons from a Crowdsourcing Experiment , 2014, COLING.

[27]  A. N. Beard.  We Don't Know What We Don't Know , 2003 .

[28]  Dong Nguyen,et al.  A Kernel Independence Test for Geographical Language Variation , 2016, CL.

[29]  Emily M. Bender,et al.  Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science , 2018, TACL.

[30]  Huan Liu,et al.  Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose , 2013, ICWSM.

[31]  William L. Hamilton,et al.  Language from police body camera footage shows racial disparities in officer respect , 2017, Proceedings of the National Academy of Sciences.

[32]  M. Williams,et al.  Towards an Ethical Framework for Publishing Twitter Data in Social Research: Taking into Account Users’ Views, Online Context and Algorithmic Estimation , 2017, Sociology.

[33]  Eric Gilbert,et al.  The Internet's Hidden Rules , 2018, Proceedings of the ACM on Human-Computer Interaction.

[34]  Emre Kıcıman,et al.  Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries , 2018, Front. Big Data.

[35]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[36]  Antske Fokkens,et al.  Offspring from Reproduction Problems: What Replication Failure Teaches Us , 2013, ACL.

[37]  J. Pennebaker,et al.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods , 2010 .

[38]  Zachary Chase Lipton.  The mythos of model interpretability , 2016, ACM Queue.

[39]  Arkaitz Zubiaga,et al.  Stance Classification in Rumours as a Sequential Task Exploiting the Tree Structure of Social Media Conversations , 2016, COLING.

[40]  Timothy R. Tangherlini.  Big Folklore: A Special Issue on Computational Folkloristics , 2016 .

[41]  D. Boyd,et al.  CRITICAL QUESTIONS FOR BIG DATA , 2012 .

[42]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[43]  D. Collier,et al.  Measurement Validity: A Shared Standard for Qualitative and Quantitative Research , 2001, American Political Science Review.

[44]  Brendan T. O'Connor,et al.  From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series , 2010, ICWSM.

[45]  Djoerd Hiemstra,et al.  #SupportTheCause: Identifying Motivations to Participate in Online Health Campaigns , 2015, EMNLP.

[46]  B. Hamm.  Projections of Power: Framing News, Public Opinion, and U.S. Foreign Policy , 2004 .

[47]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[48]  Scott Sanner,et al.  Improving LDA topic models for microblogs via tweet pooling and automatic labeling , 2013, SIGIR.

[49]  Dong Nguyen,et al.  Automatic Enrichment and Classification of Folktales in the Dutch Folktale Database , 2016 .

[50]  Dirk Hovy,et al.  Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter , 2016, NAACL.

[51]  Been Kim,et al.  Towards A Rigorous Science of Interpretable Machine Learning , 2017, 1702.08608.

[52]  Christopher M. Danforth,et al.  Characterizing the Google Books Corpus: Strong Limits to Inferences of Socio-Cultural and Linguistic Evolution , 2015, PloS one.

[53]  Timothy Baldwin,et al.  Lexical Normalisation of Short Text Messages: Makn Sens a #twitter , 2011, ACL.

[54]  Shankar Kumar,et al.  Normalization of non-standard words , 2001, Comput. Speech Lang..

[55]  Justin Grimmer,et al.  Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts , 2013, Political Analysis.

[56]  F. Mosteller,et al.  Inference in an Authorship Problem , 1963 .

[57]  P. Eckert.  Age as a Sociolinguistic Variable , 2017 .

[58]  Daniela Stockmann,et al.  We Don't Know What We Don't Know: When and How the Use of Twitter's Public APIs Biases Scientific Inference , 2017 .

[59]  Arkaitz Zubiaga,et al.  Hawkes Processes for Continuous Time Sequence Classification: an Application to Rumour Stance Classification in Twitter , 2016, ACL.

[60]  Paul DiMaggio,et al.  Adapting computational text analysis to social science (and vice versa) , 2015, Big Data Soc..

[61]  Michael Piotrowski,et al.  Natural Language Processing for Historical Texts , 2012, Synthesis Lectures on Human Language Technologies.

[62]  Inioluwa Deborah Raji,et al.  Model Cards for Model Reporting , 2018, FAT.

[63]  Christopher Potts,et al.  Enculturation Trajectories: Language, Cultural Adaptation, and Individual Outcomes in Organizations , 2015, Manag. Sci..

[64]  Andrew Piper.  Novel Devotions: Conversional Reading, Computational Modeling, and the Modern Novel , 2015 .

[65]  Djoerd Hiemstra,et al.  Predicting relevance based on assessor disagreement: analysis and practical applications for search evaluation , 2015, Information Retrieval Journal.

[66]  J. Overhage,et al.  Sorting Things Out: Classification and Its Consequences , 2001, Annals of Internal Medicine.

[67]  Michael S. Bernstein,et al.  Empath: Understanding Topic Signals in Large-Scale Text , 2016, CHI.

[68]  Simon DeDeo,et al.  Exploration and exploitation of Victorian science in Darwin’s reading notebooks , 2015, Cognition.

[69]  Dirk Hovy,et al.  Tagging Performance Correlates with Author Age , 2015, ACL.

[70]  Hau L. Lee,et al.  Socially and Environmentally Responsible Value Chain Innovations: New Operations Management Research Opportunities , 2017, Manag. Sci..

[71]  Kimberly A. Neuendorf,et al.  The Content Analysis Guidebook , 2001 .

[72]  Timnit Gebru,et al.  Datasheets for datasets , 2018, Commun. ACM.

[73]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[74]  Arthur Spirling,et al.  Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It , 2017, Political Analysis.

[75]  Yejin Choi,et al.  The Risk of Racial Bias in Hate Speech Detection , 2019, ACL.

[76]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[77]  Günter Neumann,et al.  Arabic Computational Morphology , 2007 .

[78]  David M. Mimno,et al.  Comparing Apples to Apple: The Effects of Stemmers on Topic Models , 2016, TACL.

[79]  Taku Kudo,et al.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing , 2018, EMNLP.

[80]  Diyi Yang,et al.  Seekers, Providers, Welcomers, and Storytellers: Modeling Social Roles in Online Health Communities , 2019, CHI.

[81]  Scott A. Golder,et al.  Diurnal and Seasonal Mood Vary with Work, Sleep, and Daylength Across Diverse Cultures , 2011 .

[82]  Carolyn Penstein Rosé,et al.  Computational Sociolinguistics: A Survey , 2016, Computational Linguistics.

[83]  Günter Neumann,et al.  Arabic Computational Morphology: Knowledge-based and Empirical Methods , 2007 .

[84]  D. Campbell,et al.  Unobtrusive Measures: Nonreactive Research in the Social Sciences , 1966 .

[85]  Eric P. Xing,et al.  Sparse Additive Generative Models of Text , 2011, ICML.

[86]  Cristian Danescu-Niculescu-Mizil,et al.  Conversations Gone Awry: Detecting Early Signs of Conversational Failure , 2018, ACL.

[87]  Jacob Eisenstein,et al.  You Can't Stay Here , 2017 .

[88]  Douglas G Altman,et al.  Dichotomizing continuous predictors in multiple regression: a bad idea , 2006, Statistics in medicine.

[89]  Eric P. Xing,et al.  Diffusion of Lexical Change in Social Media , 2012, PloS one.

[90]  David M. Mimno,et al.  Applications of Topic Models , 2017, Found. Trends Inf. Retr..

[91]  Michael S. Bernstein,et al.  Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions , 2017, CSCW.

[92]  Hoyt Long,et al.  Literary Pattern Recognition: Modernism between Close Reading and Machine Learning , 2016, Critical Inquiry.