Challenges in Annotation: Annotator Experiences from a Crowdsourced Emotion Annotation Task

With the prevalence of machine learning in natural language processing and other fields, an increasing number of crowd-sourced data sets are created and published. However, very little has been written about the annotation process from the point of view of the annotators. This pilot study aims to help fill the gap and provide insights into how to maximize the quality of the annotation output of crowd-sourced annotations with a focus on fine-grained sentence-level sentiment and emotion annotation from the annotators point of view.

[1]  Saif Mohammad,et al.  A Practical Guide to Sentiment Annotation: Challenges and Solutions , 2016, WASSA@NAACL-HLT.

[2]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[3]  Roman Klinger,et al.  An Analysis of Annotated Corpora for Emotion Classification in Text , 2018, COLING.

[4]  Kenton O'Hara,et al.  Gamification. using game-design elements in non-gaming contexts , 2011, CHI Extended Abstracts.

[5]  Leyland Pitt,et al.  Is it all a game? Understanding the principles of gamification , 2015 .

[6]  Juho Hamari,et al.  Social Motivations To Use Gamification: An Empirical Study Of Gamifying Exercise , 2013, ECIS.

[7]  R. Plutchik.  A GENERAL PSYCHOEVOLUTIONARY THEORY OF EMOTION , 1980 .

[8]  Fabian Groh.  Gamification : State of the Art Definition and Utilization , 2012 .

[9]  Luis von Ahn.  Games with a Purpose , 2006, Computer.

[10]  Kalina Bontcheva,et al.  Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines , 2014, LREC.

[11]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[12]  Min-Yen Kan,et al.  Perspectives on crowdsourcing annotations for natural language processing , 2012, Language Resources and Evaluation.

[13]  Jan Dirk L. Fijnheer,et al.  Gamification , 2019, Encyclopedia of Education and Information Technologies.

[14]  Sabine Bergler,et al.  CLaC and CLaC-NB: Knowledge-based and corpus-based approaches to sentiment tagging , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[15]  Yuji Matsumoto,et al.  Emotion Classification Using Massive Examples Extracted from the Web , 2008, COLING.

[16]  Boi Faltings,et al.  Sentiment Analysis Using a Novel Human Computation Game , 2012, PWNLP@ACL.

[17]  Anita Greenhill,et al.  Playing with science:gamised aspects of gamification found on the Online Citizen Science Project - Zooniverse , 2014 .

[18]  Jörg Tiedemann,et al.  Creating a Dataset for Multilingual Fine-grained Emotion-detection Using Gamification-based Annotation , 2018, WASSA@EMNLP.

[19]  Petra Saskia Bayerl,et al.  What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic Investigation , 2011, CL.

[20]  Jörg Tiedemann,et al.  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles , 2016, LREC.

[21]  P. Ekman.  Universals and cultural differences in facial expressions of emotion. , 1972 .

[22]  Katharina J. Srnka,et al.  From Words to Numbers: How to Transform Qualitative Data into Meaningful Quantitative Results , 2007 .