In What Mood Are You Today?: An Analysis of Crowd Workers' Mood, Performance and Engagement

The mood of individuals in the workplace has been well-studied due to its influence on task performance, and work engagement. However, the effect of mood has not been studied in detail in the context of microtask crowdsourcing. In this paper, we investigate the influence of one's mood, a fundamental psychosomatic dimension of a worker's behaviour, on their interaction with tasks, task performance and perceived engagement. To this end, we conducted two comprehensive studies; (i) a survey exploring the perception of crowd workers regarding the role of mood in shaping their work, and (ii) an experimental study to measure and analyze the actual impact of workers' moods in information findings microtasks. We found evidence of the impact of mood on a worker's perceived engagement through the feeling of reward or accomplishment, and we argue as to why the same impact is not perceived in the evaluation of task performance. Our findings have broad implications on the design and workflow of crowdsourcing systems.

[1]  Nico H. Frijda,et al.  Varieties of affect: Emotions and episodes, moods, and sentiments. , 1994 .

[2]  Paul A. Cairns,et al.  A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form , 2018, Int. J. Hum. Comput. Stud..

[3]  Poul E. Heegaard,et al.  Investigating Quality of Experience in the context of adaptive video streaming : findings from an experimental user study , 2013 .

[4]  Wei-Chi Tsai,et al.  Test of a model linking employee positive moods and task performance. , 2007, The Journal of applied psychology.

[5]  Karin Ackermann,et al.  The Nature Of Emotion Fundamental Questions , 2016 .

[6]  Fernando Diaz,et al.  Robust models of mouse movement on dynamic web search results pages , 2013, CIKM.

[7]  Venkatesh,et al.  Computer Technology Training in the Workplace: A Longitudinal Investigation of the Effect of Mood. , 1999, Organizational behavior and human decision processes.

[8]  Stefan Dietze,et al.  A taxonomy of microtasks on the web , 2014, HT.

[9]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[10]  Martijn H. Vastenburg,et al.  Mood measurement with Pick-A-Mood: review of current methods and design of a pictorial self-report scale , 2016 .

[11]  Elizabeth Gerber,et al.  Affect and Creative Performance on Crowdsourcing Platforms , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[12]  Peng Dai,et al.  Inserting Micro-Breaks into Crowdsourcing Workflows , 2013, HCOMP.

[13]  Neha Gupta,et al.  Modus Operandi of Crowd Workers , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[14]  Stefan Dietze,et al.  Improving learning through achievement priming in crowdsourced information finding microtasks , 2017, LAK.

[15]  Adam Tauman Kalai,et al.  A Crowd of Your Own: Crowdsourcing for On-Demand Personalization , 2014, HCOMP.

[16]  Gianluca Demartini,et al.  It's getting crowded!: how to use crowdsourcing effectively for web science research , 2016, WebSci.

[17]  F. Parmentier,et al.  Happiness increases distraction by auditory deviant stimuli. , 2016, British journal of psychology.

[18]  Michael Christian,et al.  WORK ENGAGEMENT: A QUANTITATIVE REVIEW AND TEST OF ITS RELATIONS WITH TASK AND CONTEXTUAL PERFORMANCE , 2011 .

[19]  J. Prinz.  Gut Reactions: A Perceptual Theory of Emotion , 2004 .

[20]  Donald E. Broadbent,et al.  Effect of mood on lexical decisions , 1983 .

[21]  D. Sgroi,et al.  Happiness and Productivity , 2015, Journal of Labor Economics.

[22]  Judith Redi,et al.  Modeling Task Complexity in Crowdsourcing , 2016, HCOMP.

[23]  Stefan Dietze,et al.  Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks , 2017, ACM Trans. Comput. Hum. Interact..

[24]  P J Cooper,et al.  An experimental study of the effect of mood on body size perception. , 1992, Behaviour research and therapy.

[25]  Walter S. Lasecki,et al.  Crowd Memory: Learning in the Collective , 2012, ArXiv.

[26]  Peng Dai,et al.  And Now for Something Completely Different: Improving Crowdsourcing Workflows with Micro-Diversions , 2015, CSCW.

[27]  Stefan Siersdorfer,et al.  Competitive Game Designs for Improving the Cost Effectiveness of Crowdsourcing , 2014, CIKM.

[28]  Gianluca Demartini,et al.  An Introduction to Hybrid Human-Machine Information Systems , 2017, Found. Trends Web Sci..

[29]  Nangyeon Lim,et al.  Cultural differences in emotion: differences in emotional arousal level between the East and the West , 2016, Integrative medicine research.

[30]  A. Bakker,et al.  Work engagement: An emerging concept in occupational health psychology , 2008, Bioscience trends.

[31]  Aniket Kittur,et al.  Instrumenting the crowd: using implicit behavioral measures to predict task performance , 2011, UIST.

[32]  William A. Kahn.  Psychological Conditions of Personal Engagement and Disengagement at Work , 1990 .

[33]  Gianluca Demartini,et al.  Scaling-Up the Crowd: Micro-Task Pricing Schemes for Worker Retention and Latency Improvement , 2014, HCOMP.

[34]  Elizabeth R. Tenney,et al.  Does positivity enhance work performance?: Why, when, and what we don’t know , 2016 .

[35]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[36]  Dan Cosley,et al.  Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers' Experiences in Amazon Mechanical Turk , 2016, CHI.

[37]  Richard,et al.  Motivation through the Design of Work: Test of a Theory. , 1976 .

[38]  Andrew G. Miner,et al.  State mood, task performance, and behavior at work: A within-persons approach , 2010 .

[39]  Francis Tuerlinckx,et al.  The relation between event processing and the duration of emotional experience. , 2011, Emotion.

[40]  Eric Horvitz,et al.  Why Stop Now? Predicting Worker Engagement in Online Crowdsourcing , 2013, HCOMP.

[41]  Alessandro Bozzon,et al.  Clarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing , 2017, HT.

[42]  P. Terry,et al.  Distinctions between emotion and mood , 2005 .

[43]  Stefan Dietze,et al.  Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys , 2015, CHI.

[44]  Panagiotis G. Ipeirotis,et al.  Demographics and Dynamics of Mechanical Turk Workers , 2018, WSDM.

[45]  Gabriella Kazai,et al.  An analysis of human factors and label accuracy in crowdsourcing relevance judgments , 2013, Information Retrieval.

[46]  J. Plucker,et al.  The Effect of Mood on Problem Finding in Scientific Creativity , 2016 .

[47]  David Gross-Amblard,et al.  Using Hierarchical Skills for Optimized Task Assignment in Knowledge-Intensive Crowdsourcing , 2016, WWW.

[48]  P. Schnurr,et al.  Mood: The Frame of Mind , 2011 .

[49]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[50]  D. Watson,et al.  Toward a consensual structure of mood. , 1985, Psychological bulletin.

[51]  Lizhen Cui,et al.  Efficient scheduling in crowdsourcing based on workers' mood , 2017, 2017 IEEE International Conference on Agents (ICA).

[52]  Elizabeth Gerber,et al.  Affective computational priming and creativity , 2011, CHI.

[53]  Panagiotis G. Ipeirotis,et al.  The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk , 2015, WWW.

[54]  Stephen E. Humphrey,et al.  Integrating motivational, social, and contextual work design features: a meta-analytic summary and theoretical extension of the work design literature. , 2007, The Journal of applied psychology.

[55]  Frank M. Shipman,et al.  Experiences surveying the crowd: reflections on methods, participation, and reliability , 2013, WebSci.

[56]  Aniket Kittur,et al.  CrowdScape: interactively visualizing user behavior and output , 2012, UIST.

[57]  Elizabeth Gerber,et al.  Priming for Better Performance in Microtask Crowdsourcing Environments , 2012, IEEE Internet Computing.

[58]  H. Ellis,et al.  Irrelevant thoughts, emotional mood states, and cognitive task performance , 1991, Memory & cognition.

[59]  Peter Eckersley,et al.  How Unique Is Your Web Browser? , 2010, Privacy Enhancing Technologies.

[60]  Elena Paslaru Bontas Simperl,et al.  Improving Paid Microtasks through Gamification and Adaptive Furtherance Incentives , 2015, WWW.

[61]  Alek Felstiner.  Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry , 2011 .