CrowdForge: crowdsourcing complex work

Micro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into a large population of workers around the globe to accomplish tasks in a fraction of the time and money of more traditional methods. However, such markets typically support only simple, independent tasks, such as labeling an image or judging the relevance of a search result. Here we present a general purpose framework for micro-task markets that provides a scaffolding for more complex human computation tasks which require coordination among many individuals, such as writing an article.

[1]  Henri E. Bal,et al.  Programming languages for distributed computing systems , 1989, CSUR.

[2]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[3]  O. Williamson Transaction-Cost Economics: The Governance of Contractual Relations , 1979, The Journal of Law and Economics.

[4]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[5]  Chris Callison-Burch,et al.  Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk , 2009, EMNLP.

[6]  A. V. D. Ven,et al.  Determinants of Coordination Modes within Organizations , 1976 .

[7]  David B. Skillicorn,et al.  Models and languages for parallel computation , 1998, CSUR.

[8]  JoAnne Yates,et al.  Electronic markets and electronic hierarchies , 1987, CACM.

[9]  Aniket Kittur,et al.  Harnessing the wisdom of crowds in wikipedia: quality through coordination , 2008, CSCW.

[10]  M. Diehl,et al.  Productivity loss in brainstorming groups: Toward the solution of a riddle. , 1987 .

[11]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[12]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[13]  Aniket Kittur,et al.  Coordination in collective intelligence: the role of team structure and task interdependence , 2009, CHI.

[14]  Yuan Yu,et al.  Dryad: distributed data-parallel programs from sequential building blocks , 2007, EuroSys '07.

[15]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[16]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[17]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[18]  Lydia B. Chilton,et al.  TurKit: human computation algorithms on mechanical turk , 2010, UIST.

[19]  Matthew J. Salganik,et al.  Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market , 2006, Science.

[20]  Ravi Kumar,et al.  Pig latin: a not-so-foreign language for data processing , 2008, SIGMOD Conference.

[21]  Lydia B. Chilton,et al.  TurKit: Tools for iterative tasks on mechanical turk , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[22]  Henry Mintzberg,et al.  The Structuring of Organizations , 1979 .

[23]  Adam Finkelstein,et al.  How well do line drawings depict shape , 2009, SIGGRAPH 2009.

[24]  Kevin Crowston,et al.  The interdisciplinary study of coordination , 1994, CSUR.