Human computation: a survey and taxonomy of a growing field

The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.

[1]  J. C. R. Licklider,et al.  Man-Computer Symbiosis , 1960 .

[2]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[3]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[4]  Maja J. Matarić,et al.  Designing emergent behaviors: from local interactions to collective intelligence , 1993 .

[5]  Doug Schuler,et al.  Social computing , 1994, CACM.

[6]  Douglas B. Lenat,et al.  CYC: a large-scale investment in knowledge infrastructure , 1995, CACM.

[7]  Padhraic Smyth,et al.  Knowledge Discovery and Data Mining: Towards a Unifying Framework , 1996, KDD.

[8]  Pierre Lévy,et al.  Collective Intelligence: Mankind's Emerging World in Cyberspace , 1997 .

[9]  Guy Haworth,et al.  KQQKQQ and the Kasparov-World Game , 1999, J. Int. Comput. Games Assoc..

[10]  Wendy S. Ark,et al.  At What Cost Pervasive? A Social Comuting View of Mobile Computing Systems , 1999, IBM Syst. J..

[11]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[12]  C. Alexander.  The Factory , 2000 .

[13]  Alex Kosorukoff,et al.  Human based genetic algorithm , 2001, 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat.No.01CH37236).

[14]  Daphne Koller,et al.  Support Vector Machine Active Learning with Applications to Text Classification , 2000, J. Mach. Learn. Res..

[15]  Erik T. Mueller,et al.  Open Mind Common Sense: Knowledge Acquisition from the General Public , 2002, OTM.

[16]  Timothy Chklovski,et al.  Learner: a system for acquiring commonsense knowledge by analogy , 2003, K-CAP '03.

[17]  Paul Gomme.  Iowa electronic markets , 2003 .

[18]  Jon M. Peha,et al.  PayCash: a secure efficient Internet payment system , 2003, ICEC '03.

[19]  J. Wolfers,et al.  Prediction Markets , 2003 .

[20]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[21]  David P. Anderson,et al.  Stardust@home: A Massively Distributed Public Search for Interstellar Dust in the Stardust Interstellar Dust Collector , 2005 .

[22]  Yolanda Gil,et al.  Towards Managing Knowledge Collection from Volunteer Contributors , 2005, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[23]  Timothy Chklovski,et al.  Collecting paraphrase corpora from volunteer contributors , 2005, K-CAP '05.

[24]  Luis von Ahn.  Games with a Purpose , 2006, Computer.

[25]  Manuel Blum,et al.  Verbosity: a game for collecting common-sense facts , 2006, CHI.

[26]  Craig Gentry,et al.  Secure distributed human computation , 2005, EC '05.

[27]  Eric Horvitz,et al.  Complementary computing: policies for transferring callers from dialog systems to human receptionists , 2006, User Modeling and User-Adapted Interaction.

[28]  Aniket Kittur,et al.  He says, she says: conflict and coordination in Wikipedia , 2007, CHI.

[29]  Andrew B. Whinston,et al.  Social Computing: An Overview , 2007, Commun. Assoc. Inf. Syst..

[30]  H. Lieberman.  Common Consensus : a web-based game for collecting commonsense goals , 2007 .

[31]  Jeff Howe,et al.  Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business , 2008, Human Resource Management International Digest.

[32]  Nuno Correia,et al.  A gesture based game for image tagging , 2008, CHI Extended Abstracts.

[33]  Nenghai Yu,et al.  A comprehensive human computation framework: with application to image labeling , 2008, ACM Multimedia.

[34]  Manuel Blum,et al.  reCAPTCHA: Human-Based Character Recognition via Web Security Measures , 2008, Science.

[35]  Laura A. Dabbish,et al.  Designing games with a purpose , 2008, CACM.

[36]  Luis von Ahn.  Human Computation , 2008, ICDE.

[37]  Panagiotis G. Ipeirotis,et al.  Get another label? improving data quality and data mining using multiple, noisy labelers , 2008, KDD.

[38]  Jimmy Secretan,et al.  Picbreeder: evolving pictures collaboratively online , 2008, CHI.

[39]  Schahram Dustdar,et al.  The Human-Provided Services Framework , 2008, 2008 10th IEEE Conference on E-Commerce Technology and the Fifth IEEE Conference on Enterprise Computing, E-Commerce and E-Services.

[40]  Chrysanthos Dellarocas,et al.  Harnessing Crowds: Mapping the Genome of Collective Intelligence , 2009 .

[41]  Edith Law,et al.  Input-agreement: a new mechanism for collecting data using human computation games , 2009, CHI.

[42]  Chin-Laung Lei,et al.  A crowdsourceable QoE evaluation framework for multimedia content , 2009, ACM Multimedia.

[43]  Michael I. Mandel,et al.  Evaluation of Algorithms Using Games: The Case of Music Tagging , 2009, ISMIR.

[44]  Irwin King,et al.  Mathematical Modeling of Social Games , 2009, 2009 International Conference on Computational Science and Engineering.

[45]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[46]  Irwin King,et al.  A Survey of Human Computation Systems , 2009, 2009 International Conference on Computational Science and Engineering.

[47]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[48]  Benjamin B. Bederson,et al.  Translation by iterative collaboration between monolingual users , 2010, HCOMP '10.

[49]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[50]  Olivia Buzek,et al.  Improving Translation via Targeted Paraphrasing , 2010, EMNLP.

[51]  Chris Callison-Burch,et al.  Creating Speech and Language Data With Amazon’s Mechanical Turk , 2010, Mturk@HLT-NAACL.

[52]  Jimmy J. Lin,et al.  CrowdFlow : Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility , 2010 .

[53]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[54]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[55]  Lydia B. Chilton,et al.  Exploring iterative and parallel human computation processes , 2010, HCOMP '10.

[56]  Paul N. Bennett,et al.  A report on the human computation workshop (HComp 2009) , 2010, SKDD.

[57]  Ted S. Sindlinger,et al.  Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business , 2010 .

[58]  Jeffrey P. Bigham,et al.  VizWiz: nearly real-time answers to visual questions , 2010, W4A.

[59]  Lukas Biewald.  Massive Multiplayer Human Computation for Fun, Money, and Survival , 2011, ICWE Workshops.

[60]  Benjamin B. Bederson,et al.  MonoTrans2: a new human computation system to support monolingual translation , 2011, CHI.