The future of crowd work

Paid crowd work offers remarkable opportunities for improving productivity, social mobility, and the global economy by engaging a geographically distributed workforce to complete complex tasks on demand and at scale. But it is also possible that crowd work will fail to achieve its potential, focusing on assembly-line piecework. Can we foresee a future crowd workplace in which we would want our children to participate? This paper frames the major challenges that stand in the way of this goal. Drawing on theory from organizational behavior and distributed computing, as well as direct feedback from workers, we outline a framework that will enable crowd work that is complex, collaborative, and sustainable. The framework lays out research challenges in twelve major areas: workflow, task assignment, hierarchy, real-time response, synchronous collaboration, quality control, crowds guiding AIs, AIs guiding crowds, platforms, job design, reputation, and motivation.

[1]  Chris Callison-Burch,et al.  Creating Speech and Language Data With Amazon’s Mechanical Turk , 2010, Mturk@HLT-NAACL.

[2]  Wil M. P. van der Aalst,et al.  Workflow Patterns , 2003, Distributed and Parallel Databases.

[3]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[4]  Coye Cheshire Online Trust, Trustworthiness, or Assurance? , 2011, Daedalus.

[5]  M. Kearns,et al.  An Experimental Study of the Coloring Problem on Human Subject Networks , 2006, Science.

[6]  Thomas L. Casavant,et al.  A parallel/distributed architecture for hierarchically heterogeneous web-based cooperative applications , 2001, Future Gener. Comput. Syst..

[7]  Donald A. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[8]  John Le,et al.  Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution , 2010 .

[9]  Björn Hartmann,et al.  MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture , 2012, IEEE Internet Computing.

[10]  Karim R. Lakhani,et al.  Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis , 2011, Manag. Sci..

[11]  D. Meyer,et al.  Supporting Online Material Materials and Methods Som Text Figs. S1 to S6 References Evidence for a Collective Intelligence Factor in the Performance of Human Groups , 2022 .

[12]  Paul E. Ceruzzi,et al.  When Computers Were Human , 1991, Annals of the History of Computing.

[13]  M. Six Silberman,et al.  Ethics and tactics of professional crowdwork , 2010, XRDS.

[14]  Gwo-Hshiung Tzeng,et al.  Multi-criteria task assignment in workflow management systems , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[15]  Lydia B. Chilton,et al.  Task search in a human computation market , 2010, HCOMP '10.

[16]  Michael S. Bernstein,et al.  Crowds in two seconds: enabling realtime crowd-powered interfaces , 2011, UIST.

[17]  Dana Chandler,et al.  Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets , 2012, ArXiv.

[18]  British Journal of Sociology , 1950, Nature.

[19]  Lydia B. Chilton,et al.  TurKit: human computation algorithms on mechanical turk , 2010, UIST.

[20]  James Davis,et al.  Evaluating and improving the usability of Mechanical Turk for low-income workers in India , 2010, ACM DEV '10.

[21]  Paul R. Milgrom,et al.  Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design , 1991 .

[22]  Rob Miller,et al.  VizWiz: nearly real-time answers to visual questions , 2010, UIST.

[23]  Elizabeth Gerber,et al.  Affective computational priming and creativity , 2011, CHI.

[24]  Brian Fitzgerald,et al.  Why Hackers Do What They Do: Understanding Motivation and Effort in Free/Open Source Software Projects , 2007 .

[25]  Aniket Kittur,et al.  Can you ever trust a wiki?: impacting perceived trustworthiness in wikipedia , 2008, CSCW.

[26]  Scott R. Klemmer,et al.  Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology , 2011, UIST 2011.

[27]  John Langford,et al.  CAPTCHA: Using Hard AI Problems for Security , 2003, EUROCRYPT.

[28]  Albert T. Corbett,et al.  Cognitive Tutor: Applied research in mathematics education , 2007, Psychonomic bulletin & review.

[29]  B. Shneiderman,et al.  The Reader-to-Leader Framework: Motivating Technology-Mediated Social Participation , 2009 .

[30]  J. Leon Zhao,et al.  Workflow Automation: Overview and Research Issues , 2001, Inf. Syst. Frontiers.

[31]  Vassilis Kostakos,et al.  Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems , 2008 .

[32]  G. Hertel,et al.  Voluntary Engagement in an Open Web-based Encyclopedia: Wikipedians, and Why They Do It , 2007 .

[33]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[34]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[35]  Michael Anderson,et al.  Crowdsourcing Higher Education: A Design Proposal for Distributed Learning , 2011 .

[36]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[37]  H. Rittel,et al.  Dilemmas in a general theory of planning , 1973 .

[38]  Krishna P. Gummadi,et al.  Exploring the design space of social network-based Sybil defenses , 2012, 2012 Fourth International Conference on Communication Systems and Networks (COMSNETS 2012).

[39]  Amy Bruckman,et al.  Becoming Wikipedian: transformation of participation in a collaborative online encyclopedia , 2005, GROUP.

[40]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[41]  Krzysztof Z. Gajos,et al.  Human computation tasks with global constraints , 2012, CHI.

[42]  Björn Hartmann,et al.  MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid , 2011, Human Computation.

[43]  Lydia B. Chilton,et al.  Personalized Online Education - A Crowdsourcing Challenge , 2012, HCOMP@AAAI.

[44]  Aniket Kittur,et al.  CrowdWeaver: visually managing complex crowd work , 2012, CSCW.

[45]  Timothy C. Lethbridge,et al.  Software engineering 2004 , 2006 .

[46]  Jeffrey V. Nickerson,et al.  An internet-scale idea generation system , 2013, TIIS.

[47]  Jenny Chen,et al.  Opportunities for Crowdsourcing Research on Amazon Mechanical Turk , 2011 .

[48]  Alexis Battle,et al.  The jabberwocky programming environment for structured social computing , 2011, UIST.

[49]  Kevin Crowston,et al.  The interdisciplinary study of coordination , 1994, CSUR.

[50]  David F. Redmiles,et al.  Supporting reflective practitioners , 2004, Proceedings. 26th International Conference on Software Engineering.

[51]  Edward Cutrell,et al.  mClerk: enabling mobile crowdsourcing in developing regions , 2012, CHI.

[52]  Krzysztof Z. Gajos,et al.  Platemate: crowdsourcing nutritional analysis from food photographs , 2011, UIST.

[53]  Pamela J. Hinds,et al.  Distributed Work , 2002 .

[54]  John R. Douceur,et al.  The Sybil Attack , 2002, IPTPS.

[55]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[56]  Björn Hartmann,et al.  CommunitySourcing: engaging local crowds to perform expert work via physical kiosks , 2012, CHI.

[57]  Aniket Kittur,et al.  Identifying shared leadership in Wikipedia , 2011, CHI.

[58]  David Alan Grier,et al.  Error Identification and Correction in Human Computation: Lessons from the WPA , 2011, Human Computation.

[59]  C. Lintott,et al.  Galaxy Zoo: Motivations of Citizen Scientists , 2008, 1303.6886.

[60]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[61]  Robin Milner,et al.  Communicating and mobile systems - the Pi-calculus , 1999 .

[62]  Kenton O'Hara,et al.  Social Impact , 2019, Encyclopedia of Food and Agricultural Ethics.

[63]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[64]  Tim Kraska,et al.  CrowdDB: answering queries with crowdsourcing , 2011, SIGMOD '11.

[65]  Aniket Kittur,et al.  CrowdScape: interactively visualizing user behavior and output , 2012, UIST.

[66]  Jeffrey V. Nickerson,et al.  Cooks or cobblers?: crowd creativity through combination , 2011, CHI.

[67]  S. Rafaeli,et al.  Psychological Aspects of Cyberspace: Online Motivational Factors: Incentives for Participation and Contribution in Wikipedia , 2008 .

[68]  Elif Yamangil,et al.  Towards Collaborative Intelligent Tutors: Automated Recognition of Users' Strategies , 2008, Intelligent Tutoring Systems.

[69]  Lydia B. Chilton,et al.  The labor economics of paid crowdsourcing , 2010, EC '10.

[70]  David B. Skillicorn,et al.  Models and languages for parallel computation , 1998, CSUR.

[71]  Kenton O'Hara,et al.  BISi: a blended interaction space , 2011, CHI EA '11.

[72]  Jock Given,et al.  The wealth of networks: How social production transforms markets and freedom , 2007, Inf. Econ. Policy.

[73]  Benjamin B. Bederson,et al.  Human computation: a survey and taxonomy of a growing field , 2011, CHI.

[74]  O. Williamson The Economics of Organization: The Transaction Cost Approach , 1981, American Journal of Sociology.

[75]  HighWire Press Philosophical Transactions of the Royal Society of London , 1781, The London Medical Journal.

[76]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[77]  Richard,et al.  Motivation through the Design of Work: Test of a Theory. , 1976 .

[78]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[79]  Maja Živko,et al.  Psychological Aspects of Cyberspace , 2011 .

[80]  Deborah Estrin,et al.  A hierarchical proxy architecture for Internet-scale event services , 1999, Proceedings. IEEE 8th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE'99).

[81]  Chris Callison-Burch,et al.  Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk , 2009, EMNLP.

[82]  Henry Mintzberg An Emerging Strategy of "Direct" Research , 1979 .

[83]  Alek Felstiner Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry , 2011 .

[84]  Gita Reese Sukthankar,et al.  Robust Active Learning Using Crowdsourced Annotations for Activity Recognition , 2011, Human Computation.

[85]  Luca Becchetti,et al.  Online team formation in social networks , 2012, WWW.

[86]  Praveen Paritosh,et al.  The anatomy of a large-scale human computation engine , 2010, HCOMP '10.

[87]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[88]  Aaron D. Shaw,et al.  Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India , 2012, CHI.

[89]  E. Trist The Evolution of Socio-Technical Systems: A Conceptual Framework and an Action Research Program , 1981 .

[90]  A. V. D. Ven,et al.  Determinants of Coordination Modes within Organizations , 1976 .

[91]  JoAnne Yates,et al.  Electronic markets and electronic hierarchies , 1987, CACM.

[92]  Rob Miller,et al.  Real-time crowd control of existing interfaces , 2011, UIST.

[93]  Hajo A. Reijers,et al.  Workflow Management Systems + Swarm Intelligence = Dynamic Task Assignment for Emergency Management Applications , 2007, BPM.

[94]  John C. Tang,et al.  Reflecting on the DARPA Red Balloon Challenge , 2011, Commun. ACM.

[95]  Karim R. Lakhani,et al.  Perspectives on Free and Open Source Software , 2005 .

[96]  Henri E. Bal,et al.  Programming languages for distributed computing systems , 1989, CSUR.

[97]  Aaron D. Shaw,et al.  Designing incentives for inexpert human raters , 2011, CSCW.

[98]  Neil Savage,et al.  Gaining wisdom from crowds , 2012, Commun. ACM.

[99]  Gregorio Robles,et al.  Free/Libre and Open Source Software: Survey and Study - FLOSS FINAL REPORT , 2002 .

[100]  Aniket Kittur,et al.  An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets , 2011, ICWSM.

[101]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[102]  Daren C. Brabham Moving the crowd at iStockphoto: The composition of the crowd and motivations for participation in a crowdsourcing application , 2008, First Monday.

[103]  Michael S. Bernstein,et al.  Analytic Methods for Optimizing Realtime Crowdsourcing , 2012, ArXiv.

[104]  Cristina V. Lopes,et al.  10,000 gold for 20 dollars: an exploratory study of World of Warcraft gold buyers , 2012, FDG.

[105]  Peng Dai,et al.  Artificial Intelligence for Artificial Artificial Intelligence , 2011, AAAI.

[106]  Aniket Kittur,et al.  Coordination in collective intelligence: the role of team structure and task interdependence , 2009, CHI.

[107]  Mark D. Reid,et al.  Crowd & Prejudice: An Impossibility Theorem for Crowd Labelling without a Gold Standard , 2012, ArXiv.

[108]  Dafna Shahaf,et al.  Generalized Task Markets for Human and Machine Computation , 2010, AAAI.

[109]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[110]  B. Shamir,et al.  Work-at-home and the quality of working life: some theoretical considerations , 1985 .

[111]  Michael Vitale,et al.  The Wisdom of Crowds , 2015, Cell.

[112]  Robert E. Kraut,et al.  Increasing commitment to online communities by designing for social presence , 2011, CSCW.

[113]  Eric Horvitz,et al.  Combining human and machine intelligence in large-scale crowdsourcing , 2012, AAMAS.

[114]  Wil M. P. van der Aalst,et al.  Workflow Patterns , 2004, Distributed and Parallel Databases.

[115]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[116]  Chrysanthos Dellarocas,et al.  Analyzing the economic efficiency of eBay-like online reputation reporting mechanisms , 2011, EC '01.

[117]  Lorrie Faith Cranor,et al.  Are your participants gaming the system?: screening mechanical turk workers , 2010, CHI.

[118]  Ariel D. Procaccia,et al.  Human Computation and Multiagent Systems: An Algorithmic Perspective , 2011 .

[119]  Rajkumar Buyya,et al.  A taxonomy of scientific workflow systems for grid computing , 2005, SGMD.

[120]  Gang Wang,et al.  Serf and turf: crowdturfing for fun and profit , 2011, WWW.

[121]  Finn Kensing,et al.  Participatory Design: Issues and Concerns , 2004, Computer Supported Cooperative Work (CSCW).

[122]  W. Bodmer Principles of Scientific Management , 1993, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[123]  Aytaç Göğüş Peer learning and assessment , 2012 .

[124]  Eli Biham,et al.  Proceedings of the 22nd international conference on Theory and applications of cryptographic techniques , 2003 .

[125]  Guy Haworth,et al.  KQQKQQ and the Kasparov-World Game , 1999, J. Int. Comput. Games Assoc..

[126]  Daniel Cohen-Or,et al.  Micro perceptual human computation for visual tasks , 2012, TOGS.

[127]  Paul Resnick,et al.  Trust among strangers in internet transactions: Empirical analysis of eBay' s reputation system , 2002, The Economics of the Internet and E-commerce.

[128]  S. Kerr On the folly of rewarding A, while hoping for B. , 1975, Academy of Management journal. Academy of Management.

[129]  Peng Dai,et al.  Decision-Theoretic Control of Crowd-Sourced Workflows , 2010, AAAI.

[130]  Kjeld Schmidt,et al.  Taking CSCW seriously , 1992, Computer Supported Cooperative Work (CSCW).

[131]  Cory Doctorow,et al.  For the Win , 2010 .

[132]  Bill Tomlinson,et al.  Sellers' problems in human computation markets , 2010, HCOMP '10.

[133]  A. Roth The Economist as Engineer: Game Theory, Experimentation, and Computation as Tools for Design Economics , 2002 .

[134]  Thomas P. Moran,et al.  The workaday world as a paradigm for CSCW design , 1990, CSCW '90.

[135]  Yvonne Rogers,et al.  Exploring obstacles: integrating CSCW in evolving organisations , 1994, CSCW '94.

[136]  Ohad Shamir,et al.  Vox Populi: Collecting High-Quality Labels from a Crowd , 2009, COLT.

[137]  Walter S. Lasecki,et al.  Crowd Memory: Learning in the Collective , 2012, ArXiv.

[138]  Joseph M. Hellerstein,et al.  Searching for Jim Gray: a technical overview , 2011, CACM.

[139]  J. Zittrain,et al.  Ubiquitous human computing , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[140]  James R. Glass,et al.  Collecting Voices from the Cloud , 2010, LREC.

[141]  Donald F. Roy Work Satisfaction and Social Reward in Quota Achievement: An Analysis of Piecework Incentive , 1953 .

[142]  David R. Karger,et al.  Counting with the Crowd , 2012, Proc. VLDB Endow..

[143]  Magy Seif El-Nasr,et al.  International Conference on the Foundations of Digital Games , 2012, FDG 2012.

[144]  Stacey Kuznetsov,et al.  Motivations of contributors to Wikipedia , 2006, CSOC.

[145]  Peter Druschel,et al.  Peer-to-peer systems , 2010, Commun. ACM.

[146]  Aniket Kittur,et al.  Instrumenting the crowd: using implicit behavioral measures to predict task performance , 2011, UIST.

[147]  Björn Hartmann,et al.  Turkomatic: automatic recursive task and workflow design for mechanical turk , 2011, Human Computation.

[148]  Benjamin B. Bederson,et al.  Web workers unite! addressing challenges of online laborers , 2011, CHI Extended Abstracts.

[149]  Michael S. Bernstein,et al.  4chan and /b/: An Analysis of Anonymity and Ephemerality in a Large Online Community , 2011, ICWSM.

[150]  E. Sadlon,et al.  The Karma of Digg : Reciprocity in Online Social Networks , 2009 .

[151]  Abhijit Karnik,et al.  PiVOT: personalized view-overlays for tabletops , 2012, UIST.

[152]  Aniket Kittur,et al.  Crowdsourcing, collaboration and creativity , 2010, XRDS.

[153]  Joan V. Robinson,et al.  The Nature of the Firm , 2004 .

[154]  Saul Greenberg,et al.  GroupSketch : A multi-user sketchpad for geographically-distributed small groups , 2015 .

[155]  Panagiotis G. Ipeirotis,et al.  Get another label? improving data quality and data mining using multiple, noisy labelers , 2008, KDD.

[156]  A. Pentland,et al.  Collective intelligence , 2006, IEEE Comput. Intell. Mag..

[157]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[158]  Arun Sundararajan,et al.  Optimal Design of Crowdsourcing Contests , 2009, ICIS.

[159]  Jeffrey V. Nickerson,et al.  Appropriation and Creativity: User-Initiated Contests in Scratch , 2011, 2011 44th Hawaii International Conference on System Sciences.

[160]  John E. Kelly Understanding Taylorism: Some Comments , 1978 .

[161]  Eric von Hippel,et al.  Task Partitioning: An Innovation Process Variable , 2015 .

[162]  Hiroshi Ishii,et al.  ClearBoard: a seamless medium for shared drawing and conversation with eye contact , 1992, CHI.