Sentence-Level BERT and Multi-Task Learning of Age and Gender in Social Media

Social media currently provide a window on our lives, making it possible to learn how people from different places, with different backgrounds, ages, and genders use language. In this work we exploit a newly-created Arabic dataset with ground truth age and gender labels to learn these attributes both individually and in a multi-task setting at the sentence level. Our models are based on variations of deep bidirectional neural networks. More specifically, we build models with gated recurrent units and bidirectional encoder representations from transformers (BERT). We show the utility of multi-task learning (MTL) on the two tasks and identify task-specific attention as a superior choice in this context. We also find that a single-task BERT model outperform our best MTL models on the two tasks. We report tweet-level accuracy of 51.43% for the age task (three-way) and 65.30% on the gender task (binary), both of which outperforms our baselines with a large margin. Our models are language-agnostic, and so can be applied to other languages.

[1]  Dan Goldwasser,et al.  Classification of Moral Foundations in Microblog Political Discourse , 2018, ACL.

[2]  Wiebke Bleidorn,et al.  Using Machine Learning to Advance Personality Assessment and Theory , 2019, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[3]  Benno Stein,et al.  Overview of the 3rd Author Profiling Task at PAN 2015 , 2015, CLEF.

[4]  Lyle H. Ungar,et al.  Exploring Stylistic Variation with Age and Income on Twitter , 2016, ACL.

[5]  Michael Barlow.  The individual and the group from a corpus perspective , 2018, Studies in Corpus Linguistics.

[6]  Clive Holes,et al.  Modern Arabic: Structures, Functions, and Varieties , 1996 .

[7]  Benno Stein,et al.  Overview of the 2 nd Author Profiling Task at PAN 2014 , 2014 .

[8]  Rich Caruana,et al.  Multitask Learning , 1997, Machine-mediated learning.

[9]  Kareem Darwish,et al.  Using Twitter to Collect a Multi-Dialectal Corpus of Arabic , 2014, ANLP@EMNLP.

[10]  Khaled Alrifai,et al.  Arabic Tweeps Gender and Dialect Prediction , 2017, CLEF.

[11]  Tal Hassner,et al.  Age and gender classification using convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  D. Holmes.  The Evolution of Stylometry in Humanities Scholarship , 1998 .

[13]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[14]  김혜숙,et al.  Sociolinguistics , 2004, Language Teaching.

[15]  A. Arvidsson,et al.  Echo Chamber or Public Sphere? Predicting Political Orientation and Measuring Political Homophily in Twitter Using Big Data , 2014 .

[16]  Anders Søgaard,et al.  Deep multi-task learning with low level tasks supervised at lower layers , 2016, ACL.

[17]  S. Matza,et al.  Psychological targeting as an effective approach to digital mass persuasion , 2017 .

[18]  David Yarowsky,et al.  Classifying latent user attributes in twitter , 2010, SMUC '10.

[19]  Jing Chen,et al.  Joint Learning with both Classification and Regression Models for Age Prediction , 2019, Journal of Physics: Conference Series.

[20]  D. Gritzalis,et al.  SOCIAL MEDIA PROFILING: A PANOPTICON OR OMNIOPTICON TOOL? , 2014 .

[21]  S. Argamon,et al.  Register in computational language research , 2019, Register Studies.

[22]  Benno Stein,et al.  A Decade of Shared Tasks in Digital Text Forensics at PAN , 2019, ECIR.

[23]  Diana Inkpen,et al.  Gender Identification in Twitter using N-grams and LSA: Notebook for PAN at CLEF 2018 , 2018, CLEF.

[24]  Maarten Sap,et al.  Developing Age and Gender Predictive Lexica over Social Media , 2014, EMNLP.

[25]  Malvina Nissim,et al.  Bleaching Text: Abstract Features for Cross-lingual Gender Prediction , 2018, ACL.

[26]  Richard Socher,et al.  Learned in Translation: Contextualized Word Vectors , 2017, NIPS.

[27]  Carolyn Penstein Rosé,et al.  Author Age Prediction from Text using Linear Regression , 2011, LaTeCH@ACL.

[28]  K. Versteegh.  The Arabic Language , 1997 .

[29]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[30]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[31]  Quoc V. Le,et al.  Multi-task Sequence to Sequence Learning , 2015, ICLR.

[32]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[33]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[34]  Lyle H. Ungar,et al.  Diachronic degradation of language models: Insights from social media , 2018, ACL.

[35]  Benno Stein,et al.  Overview of the Author Profiling Task at PAN 2013 , 2013, CLEF.

[36]  Walter Daelemans,et al.  TwiSty: A Multilingual Twitter Stylometry Corpus for Gender and Personality Profiling , 2016, LREC.

[37]  Lyle H. Ungar,et al.  Beyond Binary Labels: Political Ideology Prediction of Twitter Users , 2017, ACL.

[38]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[39]  Benno Stein,et al.  Overview of the 4th Author Profiling Task at PAN 2016: Cross-Genre Evaluations , 2016, CLEF.

[40]  Houda Bouamor,et al.  Fine-Grained Arabic Dialect Identification , 2018, COLING.

[41]  Soroush Vosoughi,et al.  Twitter Demographic Classification Using Deep Multi-modal Multi-task Learning , 2017, ACL.

[42]  Michael Barlow,et al.  Individual differences and usage-based grammar , 2013 .

[43]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[44]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[45]  Willibald Ruch,et al.  The language of character strengths: Predicting morally valued traits on social media , 2019, Journal of personality.

[46]  Sudeshna Sarkar,et al.  Stylometric Analysis of Bloggers' Age and Gender , 2009, ICWSM.

[47]  Nizar Habash,et al.  Introduction to Arabic Natural Language Processing , 2010, Introduction to Arabic Natural Language Processing.

[48]  Stergios Chatzikyriakidis,et al.  Shami: A Corpus of Levantine Arabic Dialects , 2018, LREC.

[49]  Adam Joinson,et al.  Human and Computer Personality Prediction From Digital Footprints , 2019, Current Directions in Psychological Science.

[50]  Reem Bassiouney.  Arabic Sociolinguistics by Reem Bassiouney , 2009 .

[51]  Michael Gamon,et al.  Linguistic correlates of style: authorship classification with deep linguistic analysis features , 2004, COLING.

[52]  Rich Caruana,et al.  Multitask Learning: A Knowledge-Based Source of Inductive Bias , 1993, ICML.

[53]  Wajdi Zaghouani,et al.  Arap-Tweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification , 2018, LREC.

[54]  William Labov,et al.  The exact description of a speech community , 1989 .

[55]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[56]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[57]  Muhammad Abdul-Mageed,et al.  You Tweet What You Speak: A City-Level Dataset of Arabic Dialects , 2018, LREC.

[58]  John D. Burger,et al.  Discriminating Gender on Twitter , 2011, EMNLP.

[59]  Chenxi Liu,et al.  Attention Correctness in Neural Image Captioning , 2016, AAAI.

[60]  Margaret L. Kern,et al.  Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach , 2013, PloS one.