Cost of Quality in Crowdsourcing

Crowdsourcing is a business model which allows practitioners to access a rather cheap and scalable workforce. However, due to loose worker-employer relationships, skill diversity of the crowd and anonymity of participants, it tends to result in lower quality compared to traditional ways of doing work. Thus crowdsourcing practitioners have to apply certain techniques to make sure that the end product complies with the quality requirements. Quality assurance techniques impact project costs and schedules. A well-defined methodology is needed to estimate these impacts in order to manage the crowdsourcing process effectively and efficiently. This paper introduces cost models of common quality assurance techniques that may be applied in crowdsourcing and describes a proposed cost of quality approach for analyzing quality related costs.

[1]  Laura A. Dabbish,et al.  Designing games with a purpose , 2008, CACM.

[2]  Kevin Crowston,et al.  From Conservation to Crowdsourcing: A Typology of Citizen Science , 2011, 2011 44th Hawaii International Conference on System Sciences.

[3]  D. Malcolm,et al.  Application of a Technique for Research and Development Program Evaluation , 1959 .

[4]  Roel Wieringa,et al.  Technical Action Research as a Validation Method in Information Systems Design Science , 2012, DESRIST.

[5]  Tobias Hoßfeld,et al.  Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms , 2013, Math. Comput. Model..

[6]  Sudhir Agarwal,et al.  Managing Quality of Human-Based eServices , 2008, ICSOC Workshops.

[7]  Phuoc Tran-Gia,et al.  Modeling of crowdsourcing platforms and granularity of work organization in Future Internet , 2011, 2011 23rd International Teletraffic Congress (ITC).

[8]  Teruaki Kitasuka,et al.  A Preliminary Study of the Number of Votes under Majority Rule in Crowdsourcing , 2013, KES.

[9]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[10]  Anne C. Rouse,et al.  A Preliminary Taxonomy of Crowdsourcing , 2010 .

[11]  Luis von Ahn,et al.  Human computation , 2009, 2009 46th ACM/IEEE Design Automation Conference.

[12]  Sylvain Arlot,et al.  A survey of cross-validation procedures for model selection , 2009, 0907.4728.

[13]  Charles E. Clark,et al.  Monte Carlo , 2006 .

[14]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[15]  Philip B. Crosby,et al.  Quality Is Free: The Art of Making Quality Certain , 1979 .

[16]  Armand V. Feigenbaum,et al.  Total quality control , 1961 .

[17]  Benjamin B. Bederson,et al.  Human computation: a survey and taxonomy of a growing field , 2011, CHI.

[18]  Hans Thies,et al.  Quality Assurance for Human-Based Electronic Services: A Decision Matrix for Choosing the Right Approach , 2010, ICWE Workshops.

[19]  John Le,et al.  Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution , 2010 .

[20]  Gabriella Kazai,et al.  Worker types and personality traits in crowdsourcing relevance labels , 2011, CIKM '11.

[21]  Martin Schader,et al.  Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes , 2011, AMCIS.

[22]  Eric Schenk,et al.  Towards a characterization of crowdsourcing practices , 2011 .

[23]  Vincent Yun Shen,et al.  Software Effort Estimation and Productivity , 1985, Adv. Comput..

[24]  Maja Vukovic,et al.  Crowdsourcing for Enterprises , 2009, 2009 Congress on Services - I.

[25]  Claudio Bartolini,et al.  Towards a Research Agenda for Enterprise Crowdsourcing , 2010, ISoLA.

[26]  Lukas Biewald,et al.  Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing , 2011, Human Computation.

[27]  Devavrat Shah,et al.  Budget-optimal crowdsourcing using low-rank matrix approximations , 2011, 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[28]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[29]  Vince Thomson,et al.  A review of research on cost of quality models and best practices , 2006 .

[30]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[31]  AnHai Doan,et al.  Matching Schemas in Online Communities: A Web 2.0 Approach , 2008, 2008 IEEE 24th International Conference on Data Engineering.

[32]  Chien-Ju Ho,et al.  Online Task Assignment in Crowdsourcing Markets , 2012, AAAI.

[33]  Elisa Bertino,et al.  Quality Control in Crowdsourcing Systems: Issues and Directions , 2013, IEEE Internet Computing.

[34]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[35]  Claude Y. Laporte,et al.  Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation , 2012 .

[36]  Panagiotis G. Ipeirotis,et al.  Get another label? improving data quality and data mining using multiple, noisy labelers , 2008, KDD.

[37]  Pietro Perona,et al.  Online crowdsourcing: Rating annotators and obtaining cost-effective labels , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.