Analyzing Biases in Human Perception of User Age and Gender from Text

User traits disclosed through written text, such as age and gender, can be used to personalize applications such as recommender systems or conversational agents. However, human perception of these traits is not perfectly aligned with reality. In this paper, we conduct a large-scale crowdsourcing experiment on guessing age and gender from tweets. We systematically analyze the quality and possible biases of these predictions. We identify the textual cues which lead to miss-assessments of traits or make annotators more or less confident in their choice. Our study demonstrates that differences between real and perceived traits are noteworthy and elucidates inaccurately used stereotypes in human perception.

[1]  Yoram Bachrach,et al.  Studying User Income through Language, Behaviour and Affect in Social Media , 2015, PloS one.

[2]  Dirk Hovy,et al.  Tagging Performance Correlates with Author Age , 2015, ACL.

[3]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[4]  Clark McCauley,et al.  Are stereotypes exaggerated? A sampling of racial, gender, academic, occupational, and political stereotypes. , 1995 .

[5]  Ingemar J. Cox,et al.  Inferring the Socioeconomic Status of Social Media Users Based on Behaviour and Language , 2016, ECIR.

[6]  T. Graepel,et al.  Private traits and attributes are predictable from digital records of human behavior , 2013, Proceedings of the National Academy of Sciences.

[7]  Russell H. Fazio,et al.  Women as Men and People: Effects of Gender-Marked Language , 1996 .

[8]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[9]  M. Kosinski,et al.  Computer-based personality judgments are more accurate than those made by humans , 2015, Proceedings of the National Academy of Sciences.

[10]  Margaret L. Kern,et al.  Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach , 2013, PloS one.

[11]  C Neil Macrae,et al.  Categorizing others: the dynamics of person construal. , 2005, Journal of personality and social psychology.

[12]  Sara Rosenthal,et al.  Age Prediction in Blogs: A Study of Style, Content, and Online Behavior in Pre- and Post-Social Media Generations , 2011, ACL.

[13]  Rafael Alonso,et al.  Extracting Social Power Relationships from Natural Language , 2011, ACL.

[14]  Benno Stein,et al.  Overview of the 3rd Author Profiling Task at PAN 2015 , 2015, CLEF.

[15]  Aron Culotta,et al.  Predicting the Demographics of Twitter Users from Website Traffic Data , 2015, AAAI.

[16]  Joseph Bates,et al.  Personality-rich believable agents that use language , 1997, AGENTS '97.

[17]  Francesco Ricci,et al.  User Personality and the New User Problem in a Context-Aware Point of Interest Recommender System , 2015, ENTER.

[18]  Dong Nguyen,et al.  "How Old Do You Think I Am?" A Study of Language and Age in Twitter , 2013, ICWSM.

[19]  John D. Burger,et al.  Discriminating Gender on Twitter , 2011, EMNLP.

[20]  Timothy Baldwin,et al.  langid.py: An Off-the-shelf Language Identification Tool , 2012, ACL.

[21]  O. John,et al.  Accuracy and bias in self-perception: individual differences in self-enhancement and the role of narcissism. , 1994, Journal of personality and social psychology.

[22]  Maarten Sap,et al.  The role of personality, age, and gender in tweeting about mental illness , 2015, CLPsych@HLT-NAACL.

[23]  Sheridan D. Blau,et al.  The gender-linked language effect in primary and secondary students' impromptu essays , 1990 .

[24]  Загоровская Ольга Владимировна,et al.  Исследование влияния пола и психологических характеристик автора на количественные параметры его текста с использованием программы Linguistic Inquiry and Word Count , 2015 .

[25]  Svitlana Volkova,et al.  On Predicting Sociodemographic Traits and Emotions from Communications in Social Networks and Their Implications to Online Self-Disclosure , 2015, Cyberpsychology Behav. Soc. Netw..

[26]  Iryna Gurevych,et al.  Can We Hide in the Web? Large Scale Simultaneous Age and Gender Author Profiling in Social Media Notebook for PAN at CLEF 2013 , 2013, CLEF.

[27]  Svitlana Volkova,et al.  Inferring Perceived Demographics from User Emotional Tone and User-Environment Emotional Contrast , 2016, ACL.

[28]  A. Eagly.  The science and politics of comparing women and men , 1995 .

[29]  Nikolaos Aletras,et al.  An analysis of the user occupational class through Twitter content , 2015, ACL.

[30]  G. Gleser,et al.  The relationship of sex and intelligence to choice of words: a normative study of verbal behavior. , 1959, Journal of clinical psychology.

[31]  David Bamman,et al.  Gender identity and lexical variation in social media , 2012, 1210.4567.

[32]  Daniel Jurafsky,et al.  Linguistic Models for Analyzing and Detecting Biased Language , 2013, ACL.

[33]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[34]  Dirk Hovy,et al.  Cross-lingual syntactic variation over age and gender , 2015, CoNLL.

[35]  Shlomo Argamon,et al.  Effects of Age and Gender on Blogging , 2006, AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs.

[36]  Mark Dredze,et al.  Quantifying Mental Health Signals in Twitter , 2014, CLPsych@ACL.

[37]  Salvatore Giorgi,et al.  Analyzing crowdsourced assessment of user traits through Twitter posts , 2015 .

[38]  Dong Nguyen,et al.  Why Gender and Age Prediction from Tweets is Hard: Lessons from a Crowdsourcing Experiment , 2014, COLING.

[39]  Maarten Sap,et al.  Developing Age and Gender Predictive Lexica over Social Media , 2014, EMNLP.

[40]  Subhagata Chattopadhyay,et al.  An Automated System to Diagnose the Severity of Adult Depression , 2011, 2011 Second International Conference on Emerging Applications of Information Technology.

[41]  Carla J. Groom,et al.  Gender Differences in Language Use: An Analysis of 14,000 Text Samples , 2008 .

[42]  D. A. Kenny,et al.  Accuracy in interpersonal perception: a social relations analysis. , 1987, Psychological bulletin.

[43]  M. Mather,et al.  Aging and motivated cognition: the positivity effect in attention and memory , 2005, Trends in Cognitive Sciences.

[44]  Saif Mohammad,et al.  CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON , 2013, Comput. Intell..

[45]  J. R. Mcmillan,et al.  Women's language: Uncertainty or interpersonal sensitivity and emotionality? , 1977 .

[46]  Lyle H. Ungar,et al.  Discovering User Attribute Stylistic Differences via Paraphrasing , 2016, AAAI.

[47]  David Yarowsky,et al.  Classifying latent user attributes in twitter , 2010, SMUC '10.

[48]  Nyla R. Branscombe,et al.  Who Considers Themselves Victims of Discrimination?: Individual Difference Predictors of Perceived Gender Discrimination in Women and Men , 1997 .

[49]  Joseph Bates,et al.  The role of emotion in believable agents , 1994, CACM.

[50]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[51]  Michael Poulin,et al.  Growing to Trust , 2015 .

[52]  Eric Horvitz,et al.  Social media as a measurement tool of depression in populations , 2013, WebSci.

[53]  J. Pennebaker,et al.  Psychological aspects of natural language. use: our words, our selves. , 2003, Annual review of psychology.

[54]  Lyle H. Ungar,et al.  Exploring Stylistic Variation with Age and Income on Twitter , 2016, ACL.

[55]  Prasant Mohapatra,et al.  Your Installed Apps Reveal Your Gender and More! , 2015, MOCO.

[56]  Brendan T. O'Connor,et al.  A Latent Variable Model for Geographic Lexical Variation , 2010, EMNLP.

[57]  Benno Stein,et al.  Overview of the 2 nd Author Profiling Task at PAN 2014 , 2014 .

[58]  Rinat B. Rosenberg-Kima,et al.  Interface agents as social models for female students: The effects of agent visual presence and appearance on female students' attitudes and beliefs , 2008, Comput. Hum. Behav..

[59]  Yanghee Kim,et al.  Pedagogical Agent Design: The Impact of Agent Realism, Gender, Ethnicity, and Instructional Role , 2004, Intelligent Tutoring Systems.

[60]  Svitlana Volkova,et al.  Inferring Latent User Properties from Texts Published in Social Media , 2015, AAAI.

[61]  Svitlana Volkova,et al.  Inferring User Political Preferences from Streaming Communications , 2014, ACL.

[62]  Trevor Cohn,et al.  Predicting and Characterising User Impact on Twitter , 2014, EACL.