Crowd-Powered Systems

Crowd-powered systems combine computation with human intelligence, drawn from large groups of people connecting and coordinating online. These hybrid systems enable applications and experiences that neither crowds nor computation could support alone.Unfortunately, crowd work is error-prone and slow, making it difficult to incorporate crowds as first-order building blocks in software. We introduce computational techniques that decompose complex tasks into simpler, verifiable steps to improve quality, and optimize work to return results in seconds. Using these techniques, we prototype a set of interactive crowd-powered systems. The first, Soylent, is a word processor that uses paid micro-contributions to aid writing tasks such as text shortening and proofreading. Using Soylent is like having access to an entire editorial staff as you write. The second system, Adrenaline, is a camera that uses crowds to help amateur photographers capture the exact right moment for a photo. It finds the best smile and catches subjects in mid-air jumps, all in realtime. These systems point to a future where social and crowd intelligence are central elements of interaction, software, and computation.

[1]  Alessandro Bozzon,et al.  Answering search queries with CrowdSearcher , 2012, WWW.

[2]  Robert E. Kraut,et al.  Motivating participation by displaying the value of contribution , 2006, CHI.

[3]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[4]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[5]  Ryen W. White,et al.  Studying the use of popular destinations to enhance web search interaction , 2007, SIGIR.

[6]  Hiroshi Ishii,et al.  ClearBoard: a seamless medium for shared drawing and conversation with eye contact , 1992, CHI.

[7]  Paul Dourish,et al.  Awareness and coordination in shared workspaces , 1992, CSCW '92.

[8]  Gerhard Weikum,et al.  WWW 2007 / Track: Semantic Web Session: Ontologies ABSTRACT YAGO: A Core of Semantic Knowledge , 2022 .

[9]  Jeffrey V. Nickerson,et al.  Cooks or cobblers?: crowd creativity through combination , 2011, CHI.

[10]  Abraham Bernstein,et al.  CrowdLang - First Steps Towards Programmable Human Computers for General Computation , 2011, Human Computation.

[11]  Cliff Lampe,et al.  A familiar face(book): profile elements as signals in an online social network , 2007, CHI.

[12]  Carolyn Watters,et al.  A field study characterizing Web-based information-seeking tasks , 2007 .

[13]  Björn Hartmann,et al.  CommunitySourcing: engaging local crowds to perform expert work via physical kiosks , 2012, CHI.

[14]  Siddhartha S. Srinivasa,et al.  People helping robots helping people: Crowdsourcing for grasping novel objects , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  熊谷 ユリヤ,et al.  James Surowiecki, 『The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations』, Random House, 5,2004, $24.95 , 2005 .

[16]  Björn Hartmann,et al.  MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid , 2011, Human Computation.

[17]  Krzysztof Z. Gajos,et al.  Human computation tasks with global constraints , 2012, CHI.

[18]  James D. Hollan,et al.  Beyond being there , 1992, CHI.

[19]  P. Markey Bystander intervention in computer-mediated communication , 2000 .

[20]  Tim Kraska,et al.  CrowdDB: answering queries with crowdsourcing , 2011, SIGMOD '11.

[21]  Edith Law,et al.  Towards Large-Scale Collaborative Planning: Answering High-Level Search Queries Using Human Computation , 2011, AAAI.

[22]  Pamela Effrein Sandstrom,et al.  Information Foraging Theory: Adaptive Interaction with Information , 2010, J. Assoc. Inf. Sci. Technol..

[23]  Eric Horvitz,et al.  Combining human and machine intelligence in large-scale crowdsourcing , 2012, AAMAS.

[24]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[25]  Aaron D. Shaw,et al.  Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India , 2012, CHI.

[26]  Aniket Kittur,et al.  He says, she says: conflict and coordination in Wikipedia , 2007, CHI.

[27]  Scott R. Klemmer,et al.  What would other programmers do: suggesting solutions to error messages , 2010, CHI.

[28]  Lydia B. Chilton,et al.  Addressing people's information needs directly in a web search result page , 2011, WWW.

[29]  Vikas Kumar,et al.  CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones , 2010, MobiSys '10.

[30]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[31]  James Begole,et al.  An Exploration into Activity-Informed Physical Advertising Using PEST , 2007, Pervasive.

[32]  Michael J. Muller,et al.  The dogear game: a social bookmark recommender system , 2007, GROUP.

[33]  Santa Clara,et al.  Research Issues in Social Computing * , 2007 .

[34]  Aniket Kittur,et al.  CrowdWeaver: visually managing complex crowd work , 2012, CSCW.

[35]  Chen-Bo Zhong,et al.  The Ethical Dangers of Deliberative Decision Making , 2011 .

[36]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1899, Outstanding Dissertations in the Computer Sciences.

[37]  Alexis Battle,et al.  The jabberwocky programming environment for structured social computing , 2011, UIST.

[38]  Ivan E. Sutherland,et al.  Sketch pad a man-machine graphical communication system , 1964, DAC.

[39]  Lydia B. Chilton,et al.  Task search in a human computation market , 2010, HCOMP '10.

[40]  Ivan Beschastnikh,et al.  Articulations of wikiwork: uncovering valued work in wikipedia through barnstars , 2008, CSCW.

[41]  F. Maxwell Harper,et al.  Social Comparisons to Motivate Contributions to an Online Community , 2007, PERSUASIVE.

[42]  K. Williams,et al.  Many Hands Make Light the Work: The Causes and Consequences of Social Loafing , 1979 .

[43]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[44]  David R. Karger,et al.  Human-powered Sorts and Joins , 2011, Proc. VLDB Endow..

[45]  Ariel D. Procaccia,et al.  Human Computation and Multiagent Systems: An Algorithmic Perspective , 2011 .

[46]  Edward Cutrell,et al.  mClerk: enabling mobile crowdsourcing in developing regions , 2012, CHI.

[47]  Krzysztof Z. Gajos,et al.  Platemate: crowdsourcing nutritional analysis from food photographs , 2011, UIST.

[48]  Lydia B. Chilton,et al.  TurKit: human computation algorithms on mechanical turk , 2010, UIST.

[49]  David R. Karger,et al.  Tie strength in question & answer on social network sites , 2012, CSCW '12.

[50]  Aditya G. Parameswaran,et al.  So who won?: dynamic max discovery with the crowd , 2012, SIGMOD Conference.

[51]  Ed H. Chi,et al.  Towards a model of understanding social search , 2008, SSM '08.

[52]  Edward Cutrell,et al.  What are you looking for?: an eye-tracking study of information usage in web search , 2007, CHI.

[53]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[54]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[55]  James A. Landay,et al.  Utility of human-computer interactions: toward a science of preference measurement , 2011, CHI.

[56]  Deborah A. Small,et al.  Sympathy and callousness: The impact of deliberative thought on donations to identifiable and statistical victims. , 2007 .

[57]  Benjamin B. Bederson,et al.  Human computation: a survey and taxonomy of a growing field , 2011, CHI.

[58]  Robert E. Kraut,et al.  Experiment 1 : Motivating Conversational Contributions Through Group Homogeneity and Individual Uniqueness , 2010 .

[59]  M. S. Ackerman,et al.  Answer Garden: a tool for growing organizational memory , 2015, COCS '90.

[60]  Daniel Marcu,et al.  Summarization beyond sentence extraction: A probabilistic approach to sentence compression , 2002, Artif. Intell..

[61]  Aaron Michael Koblin,et al.  The sheep market , 2009, C&C '09.

[62]  Mirella Lapata,et al.  Sentence Compression Beyond Word Deletion , 2008, COLING.

[63]  K. Bretonnel Cohen,et al.  Last Words: Amazon Mechanical Turk: Gold Mine or Coal Mine? , 2011, CL.

[64]  Erik T. Mueller,et al.  Open Mind Common Sense: Knowledge Acquisition from the General Public , 2002, OTM.

[65]  Lawrence K. Saul,et al.  Topic modeling of freelance job postings to monitor web service abuse , 2011, AISec '11.

[66]  Jimmy J. Lin An exploration of the principles underlying redundancy-based factoid question answering , 2007, TOIS.

[67]  Aniket Kittur,et al.  Instrumenting the crowd: using implicit behavioral measures to predict task performance , 2011, UIST.

[68]  Rob Miller,et al.  Crowdsourced Databases: Query Processing with People , 2011, CIDR.

[69]  Ravi Kumar,et al.  Pig latin: a not-so-foreign language for data processing , 2008, SIGMOD Conference.

[70]  Scott R. Klemmer,et al.  Coordinating tasks on the commons: designing for personal goals, expertise and serendipity , 2009, CHI.

[71]  Jaime Teevan,et al.  Understanding and predicting personal navigation , 2011, WSDM '11.

[72]  Michael S. Bernstein,et al.  Crowds in two seconds: enabling realtime crowd-powered interfaces , 2011, UIST.

[73]  Mirella Lapata,et al.  Models for Sentence Compression: A Comparison across Domains, Training Requirements and Evaluation Measures , 2006, ACL.

[74]  Kipling D. Williams,et al.  Interpersonal Relations and Group Processes Social Loafing: a Meta-analytic Review and Theoretical Integration , 2022 .

[75]  J. F. Kelley,et al.  An iterative design methodology for user-friendly natural language office information applications , 1984, TOIS.

[76]  Stephen Farrell,et al.  Socially augmenting employee profiles with people-tagging , 2007, UIST.

[77]  Vincent Miller New Media, Networking and Phatic Culture , 2008 .

[78]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[79]  Sofia Stamou,et al.  Queries without Clicks: Successful or Failed Searches? , 2009 .

[80]  Peter Norvig,et al.  Can Distributed Volunteers Accomplish Massive Data Analysis Tasks , 2001 .

[81]  Karen Kukich,et al.  Techniques for automatically correcting words in text , 1992, CSUR.

[82]  H. Simon,et al.  Rational choice and the structure of the environment. , 1956, Psychological review.

[83]  Martin Wattenberg,et al.  ManyEyes: a Site for Visualization at Internet Scale , 2007, IEEE Transactions on Visualization and Computer Graphics.

[84]  David Alan Grier,et al.  Error Identification and Correction in Human Computation: Lessons from the WPA , 2011, Human Computation.

[85]  Zoran Popovic,et al.  PhotoCity: training experts at large-scale image acquisition through a competitive game , 2011, CHI.

[86]  Pietro Perona,et al.  Visual Recognition with Humans in the Loop , 2010, ECCV.

[87]  Michael S. Bernstein,et al.  Personalization via friendsourcing , 2010, TCHI.

[88]  Oren Etzioni,et al.  Open Information Extraction from the Web , 2007, CACM.

[89]  Daniel Cohen-Or,et al.  Micro perceptual human computation for visual tasks , 2012, TOGS.

[90]  James Davis,et al.  CrowdSight: Rapidly Prototyping Intelligent Visual Processing Apps , 2011, Human Computation.

[91]  Alex Kosorukoff,et al.  Human based genetic algorithm , 2001, 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat.No.01CH37236).

[92]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[93]  Paul Resnick,et al.  Trust among strangers in internet transactions: Empirical analysis of eBay' s reputation system , 2002, The Economics of the Internet and E-commerce.

[94]  Benjamin B. Bederson,et al.  Web workers unite! addressing challenges of online laborers , 2011, CHI Extended Abstracts.

[95]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[96]  Turk Paul Wais,et al.  Towards Building a High-Quality Workforce with Mechanical , 2010 .

[97]  Michael S. Bernstein,et al.  4chan and /b/: An Analysis of Anonymity and Ephemerality in a Large Online Community , 2011, ICWSM.

[98]  Takeo Igarashi,et al.  As-rigid-as-possible shape manipulation , 2005, SIGGRAPH '05.

[99]  Edward Curry,et al.  Effects of Expertise Assessment on the Quality of Task Routing in Human Computation , 2013 .

[100]  Lydia B. Chilton,et al.  TurKit: Tools for iterative tasks on mechanical turk , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[101]  Eric Horvitz,et al.  SearchTogether: an interface for collaborative web search , 2007, UIST.

[102]  James R. Glass,et al.  Collecting Voices from the Cloud , 2010, LREC.

[103]  Jennifer Widom,et al.  Deco: declarative crowdsourcing , 2012, CIKM.

[104]  Eric Gilbert,et al.  Predicting tie strength with social media , 2009, CHI.

[105]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[106]  Susan T. Dumais,et al.  An Analysis of the AskMSR Question-Answering System , 2002, EMNLP.

[107]  A. Bruckman,et al.  Why Do People Write for Wikipedia? Incentives to Contribute to Open-Content Publishing , 2005 .

[108]  Rich Gold The Plenitude: Creativity, Innovation, and Making Stuff , 2007 .

[109]  John Le,et al.  Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution , 2010 .

[110]  David R. Karger,et al.  Counting with the Crowd , 2012, Proc. VLDB Endow..

[111]  Marti A. Hearst Search User Interfaces , 2009 .

[112]  S. Gosling,et al.  e-Perceptions : Personality Impressions Based on Personal Websites , 2004 .

[113]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[114]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[115]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[116]  Damon Horowitz,et al.  The anatomy of a large-scale social search engine , 2010, WWW '10.

[117]  Michael S. Bernstein,et al.  Gui --- phooey!: the case for text input , 2007, UIST.

[118]  Mark S. Granovetter The Strength of Weak Ties , 1973, American Journal of Sociology.

[119]  David R. Millen,et al.  Identity management: multiple presentations of self in facebook , 2007, GROUP.

[120]  Michael S. Bernstein,et al.  Collabio: a game for annotating people within social networks , 2009, UIST '09.

[121]  Panagiotis G. Ipeirotis,et al.  Get another label? improving data quality and data mining using multiple, noisy labelers , 2008, KDD.

[122]  Ted S. Sindlinger,et al.  Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business , 2010 .

[123]  Douglas C. Engelbart,et al.  A research center for augmenting human intellect , 1968, AFIPS Fall Joint Computing Conference.

[124]  Sarvapali D. Ramchurn,et al.  CollabMap: Augmenting Maps Using the Wisdom of Crowds , 2011, Human Computation.

[125]  Jane Li,et al.  Good abandonment in mobile and PC internet search , 2009, SIGIR.

[126]  Michael S. Bernstein,et al.  Analytic Methods for Optimizing Realtime Crowdsourcing , 2012, ArXiv.

[127]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[128]  Ohad Shamir,et al.  Vox Populi: Collecting High-Quality Labels from a Crowd , 2009, COLT.

[129]  Michael S. Bernstein,et al.  Direct answers for search queries in the long tail , 2012, CHI.

[130]  Walter S. Lasecki,et al.  Crowd Memory: Learning in the Collective , 2012, ArXiv.

[131]  Joseph M. Hellerstein,et al.  Searching for Jim Gray: a technical overview , 2011, CACM.

[132]  Luis von Ahn,et al.  Matchin: eliciting user preferences with an online game , 2009, CHI.

[133]  Mitchell M. Tseng,et al.  Design for mass personalization , 2010 .

[134]  Aniket Kittur,et al.  The polymath project: lessons from a successful online collaboration in mathematics , 2011, CHI.

[135]  James Fogarty,et al.  Amplifying community content creation with mixed initiative information extraction , 2009, CHI.

[136]  B. Latané,et al.  The Unresponsive Bystander: Why Doesn't He Help? , 1972 .

[137]  Rob Miller,et al.  Interactive Simultaneous Editing of Multiple Text Regions , 2001, USENIX ATC, General Track.

[138]  Panagiotis G. Ipeirotis Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[139]  John Riedl,et al.  Tagsplanations: explaining recommendations using tags , 2009, IUI.

[140]  C. Manski Interpreting the Predictions of Prediction Markets , 2004 .

[141]  Uri Gneezy,et al.  Deception: The Role of Consequences , 2005 .

[142]  Manuel Blum,et al.  reCAPTCHA: Human-Based Character Recognition via Web Security Measures , 2008, Science.

[143]  Björn Hartmann,et al.  Collaboratively crowdsourcing workflows with turkomatic , 2012, CSCW.

[144]  Michael J. Pazzani,et al.  A learning agent for wireless news access , 2000, IUI '00.

[145]  M. Kearns,et al.  An Experimental Study of the Coloring Problem on Human Subject Networks , 2006, Science.

[146]  Jerry Alan Fails,et al.  Interactive machine learning , 2003, IUI '03.

[147]  P. Resnick,et al.  Building Successful Online Communities: Evidence-Based Social Design , 2012 .

[148]  Andrei Broder,et al.  A taxonomy of web search , 2002, SIGF.

[149]  M. Six Silberman,et al.  Ethics and tactics of professional crowdwork , 2010, XRDS.

[150]  Rosalind W. Picard,et al.  Crowdsourcing Collective Emotional Intelligence , 2012, ArXiv.

[151]  Lydia B. Chilton,et al.  Exploring iterative and parallel human computation processes , 2010, HCOMP '10.

[152]  Rob Miller,et al.  VizWiz: nearly real-time answers to visual questions , 2010, UIST.

[153]  R. Cropanzano,et al.  Social Exchange Theory: An Interdisciplinary Review , 2005 .

[154]  Carl M. Harris,et al.  Fundamentals of queueing theory , 1975 .

[155]  Chris Callison-Burch,et al.  Creating Speech and Language Data With Amazon’s Mechanical Turk , 2010, Mturk@HLT-NAACL.

[156]  Luis Gravano,et al.  Learning to find answers to questions on the Web , 2004, TOIT.

[157]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[158]  Austin Henderson,et al.  Interaction design: beyond human-computer interaction , 2002, UBIQ.

[159]  Chrysanthos Dellarocas,et al.  Harnessing Crowds: Mapping the Genome of Collective Intelligence , 2009 .

[160]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[161]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[162]  Jaime G. Carbonell,et al.  Collaborative workflow for crowdsourcing translation , 2012, CSCW.

[163]  Anand Kulkarni,et al.  The Complexity of Crowdsourcing: Theoretical Problems in Human Computation , 2011 .

[164]  David Alan Grier,et al.  When computers were human , 2005 .

[165]  Luis von Ahn Games with a Purpose , 2006, Computer.

[166]  Peng Dai,et al.  Artificial Intelligence for Artificial Artificial Intelligence , 2011, AAAI.

[167]  B. Frey,et al.  The Cost of Price Incentives: An Empirical Analysis of Motivation Crowding-Out , 1997 .

[168]  Dan Morris,et al.  MySong: automatic accompaniment generation for vocal melodies , 2008, CHI.

[169]  Rob Miller,et al.  Real-time crowd control of existing interfaces , 2011, UIST.

[170]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[171]  Peng Dai,et al.  Decision-Theoretic Control of Crowd-Sourced Workflows , 2010, AAAI.

[172]  Michael S. Bernstein,et al.  Enhancing directed content sharing on the web , 2010, CHI.

[173]  Bill Tomlinson,et al.  Sellers' problems in human computation markets , 2010, HCOMP '10.

[174]  Panagiotis G. Ipeirotis,et al.  Estimating the Completion Time of Crowdsourced Tasks Using Survival Analysis Models , 2011 .

[175]  Amar Phanishayee,et al.  FAWN: a fast array of wimpy nodes , 2009, SOSP '09.