Managing Crowdsourced Human Computation

The tutorial covers an emerging topic of wide interest: Crowdsourcing. Specically, we cover areas of crowdsourcing related to managing structured and unstructured data in a web-related content. Many researchers and practitioners today see the great opportunity that becomes available through easily-available crowdsourcing platforms. However, most newcomers face the same questions: How can we manage the (noisy) crowds to generate high quality output? How to estimate the quality of the contributors? How can we best structure the tasks? How can we get results in small amounts of time and minimizing the necessary resources? How to setup the incentives? How should such crowdsourcing markets be setup? Their presented material will cover topics from a variety of elds, including computer science, statistics, economics, and psychology. Furthermore, the material will include real-life examples and case studies from years of experience in running and managing crowdsourcing applications in business settings.

[1]  Chrysanthos Dellarocas,et al.  Cooperation Without Enforcement? A Comparative Analysis of Litigation and Online Reputation as Quality Assurance Mechanisms , 2003, ICIS.

[2]  D. Upadhyay.  The Upside of Irrationality: The Unexpected Benefits of Defying Logic at Work and at Home , 2013 .

[3]  A. Hama.  Predictably Irrational: The Hidden Forces That Shape Our Decisions , 2010 .

[4]  A. P. Dawid,et al.  Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm , 1979 .

[5]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[6]  Robert P. W. Duin,et al.  Limits on the majority vote accuracy in classifier fusion , 2003, Pattern Analysis & Applications.

[7]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[8]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[9]  J. Stiglitz.  The market for 'lemons': quality uncertainty and the market mechanism , 2002 .

[10]  Peng Dai,et al.  Decision-Theoretic Control of Crowd-Sourced Workflows , 2010, AAAI.

[11]  Praveen Paritosh,et al.  Freebase: a collaboratively created graph database for structuring human knowledge , 2008, SIGMOD Conference.

[12]  Robert L. Leitheiser,et al.  Data quality in health care data warehouse environments , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[13]  Paul Resnick,et al.  Trust among strangers in internet transactions: Empirical analysis of eBay' s reputation system , 2002, The Economics of the Internet and E-commerce.

[14]  George A. Akerlof.  The Market for “Lemons”: Quality Uncertainty and the Market Mechanism , 1970 .

[15]  Lydia B. Chilton,et al.  TurKit: Tools for iterative tasks on mechanical turk , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[16]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[17]  Luis von Ahn.  Games with a Purpose , 2006, Computer.

[18]  G. Kalyanaram,et al.  Nudge: Improving Decisions about Health, Wealth, and Happiness , 2011 .

[19]  Panagiotis G. Ipeirotis,et al.  Get another label? improving data quality and data mining using multiple, noisy labelers , 2008, KDD.

[20]  Bryce Glass,et al.  Building Web Reputation Systems , 2010 .

[21]  Sudhir Agarwal,et al.  Managing Quality of Human-Based eServices , 2008, ICSOC Workshops.

[22]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.