Chatterbox: Conversational Interfaces for Microtask Crowdsourcing

Conversational interfaces can facilitate human-computer interactions. Whether or not conversational interfaces can improve worker experience and work quality in crowdsourcing marketplaces has remained unanswered. We investigate the suitability of text-based conversational interfaces for microtask crowdsourcing. We designed a rigorous experimental campaign aimed at gauging the interest and acceptance by crowdworkers for this type of work interface. We compared Web and conversational interfaces for five common microtask types and measured the execution time, quality of work, and the perceived satisfaction of 316 workers recruited from the FigureEight platform. We show that conversational interfaces can be used effectively for crowdsourcing microtasks, resulting in a high satisfaction from workers, and without having a negative impact on task execution time or work quality.

[1]  Manfred Tscheligi,et al.  SIG: Chatbots for Social Good , 2018, CHI Extended Abstracts.

[2]  Sandeep Avula,et al.  Searchbots: Using Chatbots in Collaborative Information-seeking Tasks , 2017, SIGIR.

[3]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[4]  Victor Zue,et al.  Conversational interfaces: advances and challenges , 1997, Proceedings of the IEEE.

[5]  James Davis,et al.  Evaluating and improving the usability of Mechanical Turk for low-income workers in India , 2010, ACM DEV '10.

[6]  Lei Li,et al.  CI-Bot: A Hybrid Chatbot Enhanced by Crowdsourcing , 2017, APWeb/WAIM Workshops.

[7]  Fabio Casati,et al.  CrowdCafe - Mobile Crowdsourcing Platform , 2016, ArXiv.

[8]  Deepak Ganesan,et al.  mCrowd: a platform for mobile crowdsourcing , 2009, SenSys '09.

[9]  G. Michael Youngblood,et al.  Wallah: design and evaluation of a task-centric mobile-based crowdsourcing platform , 2014, MobiQuitous.

[10]  Rich Ling,et al.  Microcoordination 2.0: Social Coordination in the Age of Smartphones and Messaging Apps , 2016 .

[11]  Sriganesh Madhvanath,et al.  Mobile Interfaces for Crowdsourced Multimedia Microtasks , 2015, IndiaHCI.

[12]  Amos Azaria,et al.  InstructableCrowd: Creating IF-THEN Rules via Conversations with the Crowd , 2016, CHI Extended Abstracts.

[13]  David Gross-Amblard,et al.  Using Hierarchical Skills for Optimized Task Assignment in Knowledge-Intensive Crowdsourcing , 2016, WWW.

[14]  Stefan Dietze,et al.  A taxonomy of microtasks on the web , 2014, HT.

[15]  Alessandro Bozzon,et al.  Human-Aided Bots , 2018, IEEE Internet Computing.

[16]  Robert G. Capra,et al.  SearchBots: User Engagement with ChatBots during Collaborative Search , 2018, CHIIR.

[17]  Alessandro Bozzon,et al.  Reactive crowdsourcing , 2013, WWW.

[18]  Amos Azaria,et al.  "Is There Anything Else I Can Help You With?" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent , 2016, HCOMP.

[19]  Frank M. Shipman,et al.  Experiences surveying the crowd: reflections on methods, participation, and reliability , 2013, WebSci.

[20]  Ranjitha Kumar,et al.  Designing the Future of Personal Fashion , 2018, CHI.

[21]  M. van Eeuwen,et al.  Mobile conversational commerce: messenger chatbots as the next interface between businesses and consumers , 2017 .

[22]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[23]  Ting-Hao Huang,et al.  Guardian: A Crowd-Powered Spoken Dialog System for Web APIs , 2015, HCOMP.

[24]  Jaime Teevan,et al.  Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop , 2017, CHI.

[25]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[26]  Björn Hartmann,et al.  MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid , 2011, Human Computation.

[27]  Charles L. A. Clarke,et al.  Exploring Conversational Search With Humans, Assistants, and Wizards , 2017, CHI Extended Abstracts.

[28]  Ting-Hao Huang,et al.  Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Time , 2018, CHI.

[29]  Eddy Maddalena,et al.  Mobile crowdsourcing: four experiments on platforms and tasks , 2014, Distributed and Parallel Databases.

[30]  Alessandro Bozzon,et al.  Clarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing , 2017, HT.