Sellers' problems in human computation markets

"Tools for human computers" is an underexplored design space in human computation research, which has focused on techniques for buyers of human computation rather than sellers. We characterize the sellers in one human computation market, Mechanical Turk, and describe some of the challenges they face. We list several projects developed to approach these problems, and conclude with a list of open questions relevant to sellers, buyers, and researchers.

[1]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[2]  Robert E. Kraut,et al.  Why pay?: exploring how financial incentives are used for question & answer , 2010, CHI.

[3]  David C. Parkes,et al.  The role of game theory in human computation systems , 2009, HCOMP '09.

[4]  Zhen Li,et al.  Botnet Economics: Uncertainty Matters , 2008, WEIS.

[5]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[6]  Andrew B. Whinston,et al.  The Dynamics of the Electronic Market: An Evolutionary Game Approach , 2000, Inf. Syst. Frontiers.

[7]  Paul Resnick,et al.  The value of reputation on eBay: A controlled experiment , 2002 .

[8]  Scott Counts,et al.  mimir: a market-based real-time question and answer service , 2009, CHI.

[9]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[10]  Panagiotis G. Ipeirotis.  Demographics of Mechanical Turk , 2010 .

[11]  K. Steiglitz.  Snipers, Shills, and Sharks: eBay and Human Behavior , 2007 .

[12]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[13]  Lydia B. Chilton,et al.  The labor economics of paid crowdsourcing , 2010, EC '10.

[14]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[15]  J. Zittrain,et al.  Ubiquitous human computing , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[16]  Jungpil Hahn,et al.  The dynamics of mass online marketplaces: a case study of an online auction , 2001, CHI.

[17]  Lorrie Faith Cranor,et al.  Are your participants gaming the system?: screening mechanical turk workers , 2010, CHI.

[18]  Desney S. Tan,et al.  Hidden markets: UI design for a P2P backup application , 2010, CHI.

[19]  Stefan Savage,et al.  An inquiry into the nature and causes of the wealth of internet miscreants , 2007, CCS '07.

[20]  Yan Chen,et al.  Re-examining price as a predictor of answer quality in an online q&a site , 2010, CHI.

[21]  Rahul Sami,et al.  A reputation system for selling human computation , 2009, HCOMP '09.

[22]  Y. Namestnikov,et al.  The economics of botnets , 2009 .

[23]  Donghui Feng,et al.  Acquiring High Quality Non-Expert Knowledge from On-Demand Workforce , 2009, PWNLP@IJCNLP.

[24]  John Joseph Horton,et al.  The Condition of the Turking Class: Are Online Employers Fair and Honest? , 2010, ArXiv.

[25]  Robert E. Kraut,et al.  Can markets help?: applying market mechanisms to improve synchronous communication , 2008, CSCW.