Subcontracting Microwork

Mainstream crowdwork platforms treat microtasks as indivisible units; however, in this article, we propose that there is value in re-examining this assumption. We argue that crowdwork platforms can improve their value proposition for all stakeholders by supporting subcontracting within microtasks. After describing the value proposition of subcontracting, we then define three models for microtask subcontracting: real-time assistance, task management, and task improvement, and reflect on potential use cases and implementation considerations associated with each. Finally, we describe the outcome of two tasks on Mechanical Turk meant to simulate aspects of subcontracting. We reflect on the implications of these findings for the design of future crowd work platforms that effectively harness the potential of subcontracting workflows.

[1]  Michael S. Bernstein,et al.  Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-internships , 2016, CHI.

[2]  Panagiotis G. Ipeirotis Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[3]  Nicholas J. Belkin,et al.  Helping people find what they don't know , 2000, CACM.

[4]  David B. Martin,et al.  TurkBench: Rendering the Market for Turkers , 2015, CHI.

[5]  Michael S. Bernstein,et al.  Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms , 2016, CSCW.

[6]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[7]  Dan Cosley,et al.  Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers' Experiences in Amazon Mechanical Turk , 2016, CHI.

[8]  J. C. Flanagan Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE , 2022 .

[9]  Björn Hartmann,et al.  Collaboratively crowdsourcing workflows with turkomatic , 2012, CSCW.

[10]  Jacki O'Neill,et al.  Being a turker , 2014, CSCW.

[11]  Vandana Singh,et al.  Open Source Technical Support: A Look at Peer Help-Giving , 2006, Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06).

[12]  Wei Wu,et al.  Structuring, Aggregating, and Evaluating Crowdsourced Design Critique , 2015, CSCW.

[13]  Jacob O. Wobbrock,et al.  LemonAid: selection-based crowdsourced contextual help for web applications , 2012, CHI.

[14]  Benjamin B. Bederson,et al.  Web workers unite! addressing challenges of online laborers , 2011, CHI Extended Abstracts.

[15]  Meredith Ringel Morris,et al.  Gauging Receptiveness to Social Microvolunteering , 2015, CHI.

[16]  Matthew Lease,et al.  Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms , 2015 .

[17]  Aniket Kittur,et al.  CrowdScape: interactively visualizing user behavior and output , 2012, UIST.

[18]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[19]  Mary L. Gray,et al.  The Crowd is a Collaborative Network , 2016, CSCW.

[20]  Amy Bruckman,et al.  Redistributing leadership in online creative collaboration , 2013, CSCW.

[21]  J. Panel Convention on the Rights of Persons with Disabilities , 2013 .

[22]  Eric A. von Hippel,et al.  How Open Source Software Works: 'Free' User-to-User Assistance? , 2000 .

[23]  Bernadette McSherry,et al.  The United Nations Convention on the rights of persons with disabilities. , 2008, Journal of law and medicine.

[24]  Aniket Kittur,et al.  CrowdWeaver: visually managing complex crowd work , 2012, CSCW.

[25]  Meredith Ringel Morris,et al.  What do people ask their social networks, and why?: a survey study of status message q&a behavior , 2010, CHI.

[26]  Meredith Ringel Morris,et al.  Accessible Crowdwork?: Understanding the Value in and Challenge of Microtask Employment for People with Disabilities , 2015, CSCW.

[27]  M. Six Silberman,et al.  Turkopticon: interrupting worker invisibility in amazon mechanical turk , 2013, CHI.

[28]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[29]  Rob Miller,et al.  Real-time crowd control of existing interfaces , 2011, UIST.

[30]  Anne Marie Piper,et al.  “Why would anybody do this?”: Older Adults’ Understanding of and Experiences with Crowd Work , 2016 .

[31]  Hironobu Takagi,et al.  Age-Based Task Specialization for Crowdsourced Proofreading , 2013, HCI.

[32]  長瀬 修 わかりやすい障害者の権利条約 : 知的障害のある人の権利のために = Convention on the rights of persons with disabilities , 2009 .

[33]  Guan Wang,et al.  Crowdsourcing from Scratch: A Pragmatic Experiment in Data Collection by Novice Requesters , 2015, HCOMP.

[34]  Michael S. Bernstein,et al.  Daemo: A Self-Governed Crowdsourcing Marketplace , 2015, UIST.

[35]  Janine Berg,et al.  Income Security in the On-Demand Economy: Findings and Policy Lessons from a Survey of Crowdworkers , 2016 .

[36]  Chris Callison-Burch,et al.  Crowd-Workers: Aggregating Information Across Turkers to Help Them Find Higher Paying Work , 2014, HCOMP.

[37]  D. Clery Galaxy evolution. Galaxy zoo volunteers share pain and glory of research. , 2011, Science.

[38]  Chris Callison-Burch,et al.  Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk , 2009, EMNLP.

[39]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.