Understanding Worker Moods and Reactions to Rejection in Crowdsourcing

Requesters on crowdsourcing platforms typically exercise the power to decide the fate of tasks completed by crowd workers. Rejecting work has a direct impact on workers; (i) they may not be rewarded for the work completed and for their effort that has been exerted, and (ii) rejection affects worker reputation and may limit their access to future work opportunities. This paper presents a comprehensive study that aims to understand worker moods and how workers react to rejections in microtask crowdsourcing. We experimentally investigate the effect of the mood of workers on their performance, and the interaction of their moods with their reactions to rejection. Finally, we explore techniques such as presenting social comparative explanations to foster positive reactions to rejection. We found that workers in pleasant moods significantly outperform those in unpleasant moods. Workers whose work is rejected due to narrowly failing pre-screening tests exhibited the most negative emotional responses.

[1]  Lei Han,et al.  All Those Wasted Hours: On Task Abandonment in Crowdsourcing , 2019, WSDM.

[2]  R. Wood.  Task complexity: Definition of the construct , 1986 .

[3]  Markus Groth,et al.  IMPROVING APPLICANTS' REACTIONS TO REJECTION LETTERS: AN APPLICATION OF FAIRNESS THEORY , 2001 .

[4]  M. Zvolensky,et al.  Emotional avoidance: an experimental test of individual differences and response suppression using biological challenge. , 2003, Behaviour research and therapy.

[5]  The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk , 2015, WWW.

[6]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[7]  Svitlana Volkova,et al.  Separating Facts from Fiction: Linguistic Models to Classify Suspicious and Trusted News Posts on Twitter , 2017, ACL.

[8]  Stefan Dietze,et al.  Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection , 2018, Computer Supported Cooperative Work (CSCW).

[9]  Michael S. Bernstein,et al.  We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers , 2015, CHI.

[10]  G. Downey,et al.  Implications of rejection sensitivity for intimate relationships. , 1996, Journal of personality and social psychology.

[11]  Kathleen J. Krone,et al.  Characteristics of Rejection Letters and their Effects on Job Applicants , 1984 .

[12]  D. Watson,et al.  Toward a consensual structure of mood. , 1985, Psychological bulletin.

[13]  Michael S. Bernstein,et al.  Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms , 2016, UIST.

[14]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[15]  J. Pennebaker,et al.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods , 2010 .

[16]  Bram Vanderborght,et al.  Design and evaluation of a DIY construction system for educational robot kits , 2016 .

[17]  Ryan L. Boyd,et al.  The Development and Psychometric Properties of LIWC2015 , 2015 .

[18]  Daniel S Pine,et al.  Rejection sensitivity and disruption of attention by social threat cues. , 2009, Journal of research in personality.

[19]  Jaime Teevan,et al.  Chain Reactions: The Impact of Order on Microtask Chains , 2016, CHI.

[20]  M. Six Silberman,et al.  Turkopticon: interrupting worker invisibility in amazon mechanical turk , 2013, CHI.

[21]  Poul E. Heegaard,et al.  Investigating Quality of Experience in the context of adaptive video streaming : findings from an experimental user study , 2013 .

[22]  M. Bradley,et al.  Remembering pictures: pleasure and arousal in memory. , 1992, Journal of experimental psychology. Learning, memory, and cognition.

[23]  Lorrie Faith Cranor,et al.  Are your participants gaming the system?: screening mechanical turk workers , 2010, CHI.

[24]  P. Schnurr,et al.  Mood: The Frame of Mind , 2011 .

[25]  T. Gilovich,et al.  Heuristics and Biases: When Less Is More: Counterfactual Thinking and Satisfaction among Olympic Medalists , 2002 .

[26]  Ujwal Gadiraju,et al.  In What Mood Are You Today?: An Analysis of Crowd Workers' Mood, Performance and Engagement , 2019, WebSci.

[27]  Tim M. Blackburn,et al.  Publication rejection among ecologists , 2003 .

[28]  N. Roese,et al.  What Might Have Been: The Social Psychology of Counterfactual Thinking , 1995 .

[29]  Dan Cosley,et al.  Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers' Experiences in Amazon Mechanical Turk , 2016, CHI.

[30]  Steven A. Murphy,et al.  The Happy-Productive Worker Thesis Revisited , 2008 .

[31]  Karen L Woolley,et al.  Handling manuscript rejection: insights from evidence and experience. , 2009, Chest.

[32]  Ricardo Kawase,et al.  Training Workers for Improving Performance in Crowdsourcing Microtasks , 2015, EC-TEL.

[33]  Sebastian Möller,et al.  Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd , 2017, Crowdsourcing and Human-Centered Experiments.

[34]  Jon D. Morris.  Observations: SAM: The Self-Assessment Manikin An Efficient Cross-Cultural Measurement Of Emotional Response 1 , 1995 .

[35]  Martijn H. Vastenburg,et al.  Mood measurement with Pick-A-Mood: review of current methods and design of a pictorial self-report scale , 2016 .

[36]  Brian T. Pentland,et al.  Task Complexity: Extending a Core Concept , 2014 .

[37]  Lukas Biewald,et al.  Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing , 2011, Human Computation.

[38]  Ting-Hao Huang,et al.  Guardian: A Crowd-Powered Spoken Dialog System for Web APIs , 2015, HCOMP.

[39]  Eshkol Rafaeli,et al.  The rejection-rage contingency in borderline personality disorder. , 2011, Journal of abnormal psychology.

[40]  J. Prinz.  Gut Reactions: A Perceptual Theory of Emotion , 2004 .

[41]  Francis Tuerlinckx,et al.  The relation between event processing and the duration of emotional experience. , 2011, Emotion.

[42]  R. Byrne.  Mental models and counterfactual thoughts about what might have been , 2002, Trends in Cognitive Sciences.

[43]  Judith Redi,et al.  Modeling Task Complexity in Crowdsourcing , 2016, HCOMP.

[44]  Stefan Dietze,et al.  Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks , 2017, ACM Trans. Comput. Hum. Interact..

[45]  Michael S. Bernstein,et al.  PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance , 2015, L@S.

[46]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[47]  Randolph C. Arnau,et al.  Pain and Emotion: Effects of Affective Picture Modulation , 2001, Psychosomatic medicine.

[48]  P. Ekman,et al.  The nature of emotion: Fundamental questions. , 1994 .

[49]  P. Terry,et al.  Distinctions between emotion and mood , 2005 .

[50]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[51]  Nico H. Frijda,et al.  Varieties of affect: Emotions and episodes, moods, and sentiments. , 1994 .

[52]  Eunsol Choi,et al.  Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking , 2017, EMNLP.

[53]  Karin Ackermann,et al.  The Nature Of Emotion Fundamental Questions , 2016 .

[54]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[55]  N. Roese.  Counterfactual thinking. , 1997, Psychological bulletin.