EmotionExpert: Facebook game for crowdsourcing annotations for emotion detection

The current paper explores the use of the social network platform Facebook, as a source of emotion annotated textual data as well as a source of annotators. The traditional approach of hiring experts to provide manually labeled (annotated) data for NLP research is time-consuming, tedious and expensive. Hence, crowdsourcing has emerged as a useful method for obtaining annotated data for natural language processing (NLP) research. We have developed a purposeful innovative Facebook game called EmotionExpert in order to collect human annotated textual data for emotion detection from text. The game provides a means to reach a large number of players, while making the annotation of emotional content of texts an enjoyable and social activity. The findings reported in this paper indicate that EmotionExpert is a useful resource for reaching a large number of people to produce reliable annotations.

[1]  Vikas Sindhwani,et al.  Data Quality from Crowdsourcing: A Study of Annotation Selection Criteria , 2009, HLT-NAACL 2009.

[2]  Chris Callison-Burch,et al.  Creating Speech and Language Data With Amazon’s Mechanical Turk , 2010, Mturk@HLT-NAACL.

[3]  Alice H. Oh,et al.  Do You Feel What I Feel? Social Aspects of Emotions in Twitter Conversations , 2012, ICWSM.

[4]  R. D'Andrade,et al.  the colors of emotion1 , 1974 .

[5]  Aki Järvinen,et al.  Game design for social networks: interaction design for playful dispositions , 2009, Sandbox@SIGGRAPH.

[6]  P. Shaver,et al.  Emotion knowledge: further exploration of a prototype approach. , 1987, Journal of personality and social psychology.

[7]  Christine Storm,et al.  A taxonomic study of the vocabulary of emotions. , 1987 .

[8]  Saif Mohammad,et al.  CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON , 2013, Comput. Intell..

[9]  Stuart Adam Battersby,et al.  Experimenting with Distant Supervision for Emotion Classification , 2012, EACL.

[10]  Laurel D. Riek,et al.  Guess What? A Game for Affective Annotation of Video Using Crowd Sourcing , 2011, ACII.

[11]  Stan Szpakowicz,et al.  Identifying Expressions of Emotion in Text , 2007, TSD.

[12]  Ted S. Sindlinger,et al.  Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business , 2010 .

[13]  Manuel Blum,et al.  Verbosity: a game for collecting common-sense facts , 2006, CHI.

[14]  Rada Mihalcea,et al.  Building a Sense Tagged Corpus with Open Mind Word Expert , 2002, SENSEVAL.

[15]  David Vickrey,et al.  Online Word Games for Semantic Data Collection , 2008, EMNLP.

[16]  Carlo Strapparava,et al.  SemEval-2007 Task 14: Affective Text , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[17]  Akshi Kumar,et al.  Sentiment Analysis: A Perspective on its Past, Present and Future , 2012 .

[18]  Roddy Cowie,et al.  What a neural net needs to know about emotion words , 1999 .

[19]  Min-Yen Kan,et al.  Perspectives on crowdsourcing annotations for natural language processing , 2012, Language Resources and Evaluation.

[20]  Roddy Cowie,et al.  Describing the emotional states that are expressed in speech , 2003, Speech Commun..

[21]  K. Scherer Psychological models of emotion. , 2000 .

[22]  Stefanie Nowak,et al.  How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation , 2010, MIR '10.

[23]  Mark Steyvers,et al.  Identifying Emotions, Intentions, and Attitudes in Text Using a Game with a Purpose , 2010, HLT-NAACL 2010.

[24]  Erik T. Mueller,et al.  Open Mind Common Sense: Knowledge Acquisition from the General Public , 2002, OTM.

[25]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[26]  Cecilia Ovesdotter Alm,et al.  Emotions from Text: Machine Learning for Text-based Emotion Prediction , 2005, HLT.

[27]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[28]  Hugo Liu,et al.  A Corpus-based Approach to Finding Happiness , 2006, AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs.

[29]  J. Russell,et al.  Concept of Emotion Viewed From a Prototype Perspective , 1984 .

[30]  Laura A. Dabbish,et al.  Designing games with a purpose , 2008, CACM.

[31]  Janne Paavilainen,et al.  The many faces of sociability and social play in games , 2009, MindTrek '09.