A profile-aware microtasking approach for improving task assignment in crowdsourcing services

Microtasking as one of the crowdsourcing models has found penetration mostly in developed nations and is slowly making inroads in developing nations. It has been used for quickly solving a myriad of social and business challenges by tapping into the diversity of the unknown crowd. Nevertheless, there are still a number of research challenges that are found in various microtasking initiatives, such as lack of user profiling, quality of task submissions, poor task design and assignment, ambiguous task assessments, and lack of diverse platforms to cater for the needs of differently skilled crowds. Most prominent microtasking platforms do not consider or even exploit the profile of the crowd in improving task design, assignment, and eventually quality of completed microtasks. In this paper, the objective was to explore the design and evaluation of a conceptual approach that could be implemented in microtasking environments for purposes of improving task assignments and eventually quality. A design science research methodology was adopted to build the proposed artefact, which was evaluated for relevance using comparative analysis and crowdsourcing metrics. The results suggest that exploiting micro workers' profiles could improve task design, assignment, assessment, and ultimately quality. However, stakeholders' awareness of effective microtasking approaches still requires research attention.

[1]  Nathan Eagle,et al.  txteagle: Mobile Crowdsourcing , 2009, HCI.

[2]  Roman Beck,et al.  Combining Design Science and Design Research Perspectives--Findings of Three Prototyping Projects , 2012, 2012 45th Hawaii International Conference on System Sciences.

[3]  Licia Capra,et al.  Quality control for real-time ubiquitous crowdsourcing , 2011, UbiCrowd '11.

[4]  Mahmood Hosseini,et al.  The four pillars of crowdsourcing: A reference model , 2014, 2014 IEEE Eighth International Conference on Research Challenges in Information Science (RCIS).

[5]  Björn Hartmann,et al.  MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture , 2012, IEEE Internet Computing.

[6]  Edward Cutrell,et al.  mClerk: enabling mobile crowdsourcing in developing regions , 2012, CHI.

[7]  Samir Chatterjee,et al.  A Design Science Research Methodology for Information Systems Research , 2008 .

[8]  Björn Hartmann,et al.  MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid , 2011, Human Computation.

[9]  Legand Burge,et al.  The potential benefits of mobile microwork services in developing nations: Research opportunities and challenges , 2014, 2014 IST-Africa Conference Proceedings.

[10]  André van der Hoek,et al.  Crowdsourcing in Software Engineering: Models, Motivations, and Challenges , 2016, IEEE Software.

[11]  G Turk,et al.  The Mechanical Turk , 2015 .

[12]  Atreyi Kankanhalli,et al.  Leveraging Crowdsourcing for Organizational Value Co-Creation , 2013, Commun. Assoc. Inf. Syst..

[13]  Shari Lawrence Pfleeger Useful Cybersecurity Metrics , 2009, IT Professional.

[14]  Christian Wagner,et al.  Participation in Micro-task Crowdsourcing Markets as Work and Leisure: The Impact of Motivation and Micro-time Structuring , 2014 .

[15]  Elizabeth Gerber,et al.  Priming for Better Performance in Microtask Crowdsourcing Environments , 2012, IEEE Internet Computing.

[16]  Elena Simperl,et al.  How to Use Crowdsourcing Effectively: Guidelines and Examples , 2015 .

[17]  Ron Weber,et al.  Research Commentary: Information Systems and Conceptual Modeling - A Research Agenda , 2002, Inf. Syst. Res..

[18]  Phuoc Tran-Gia,et al.  Series Human Cloud as Emerging Internet Application-Anatomy of the Microworkers Crowdsourcing Platform , 2011 .

[19]  Muhammad Asif,et al.  A Case of Mobile App Reviews as a Crowdsource , 2015 .

[20]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[21]  Matthias Thimm,et al.  Microtask Available, Send us your CV! , 2013, 2013 International Conference on Cloud and Green Computing.

[22]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[23]  Jabu Mtsweni,et al.  The adoption of crowdsourcing platforms in South Africa , 2015, 2015 IST-Africa Conference.

[24]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[25]  Kieran Conboy,et al.  Measuring the crowd: a preliminary taxonomy of crowdsourcing metrics , 2015, OpenSym.