Automated Workflow Synthesis

By coordinating efforts from humans and machines, human computation systems can solve problems that machines cannot tackle alone. A general challenge is to design efficient human computation algorithms or workflows with which to coordinate the work of the crowd. We introduce a method for automated workflow synthesis aimed at ideally harnessing human efforts by learning about the crowd's performance on tasks and synthesizing an optimal workflow for solving a problem. We present experimental results for human sorting tasks, which demonstrate both the benefit of understanding and optimizing the structure of workflows based on observations. Results also demonstrate the benefits of using value of information to guide experiments for identifying efficient workflows with fewer experiments.

[1]  Lydia B. Chilton,et al.  TurKit: human computation algorithms on mechanical turk , 2010, UIST.

[2]  Dafna Shahaf,et al.  Generalized Task Markets for Human and Machine Computation , 2010, AAAI.

[3]  David R. Karger,et al.  Human-powered Sorts and Joins , 2011, Proc. VLDB Endow..

[4]  Douglas R. Smith,et al.  Top-Down Synthesis of Divide-and-Conquer Algorithms , 1985, Artif. Intell..

[5]  Ronald A. Howard,et al.  Decision analysis: Perspectives on inference, decision, and experimentation , 1970 .

[6]  Krzysztof Z. Gajos,et al.  Platemate: crowdsourcing nutritional analysis from food photographs , 2011, UIST.

[7]  David A. Padua,et al.  Optimizing Sorting with Machine Learning Algorithms , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[8]  Edith Law,et al.  Human Computation , 2011, Human Computation.

[9]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[10]  Stuart J. Russell,et al.  Principles of Metareasoning , 1989, Artif. Intell..

[11]  Rob Miller,et al.  Crowdsourced Databases: Query Processing with People , 2011, CIDR.

[12]  Neoklis Polyzotis,et al.  Max algorithms in crowdsourcing environments , 2012, WWW.

[13]  Eric Horvitz,et al.  Reflection and Action Under Scarce Resources: Theoretical Principles and Empirical Study , 1989, IJCAI.

[14]  Peng Dai,et al.  Artificial Intelligence for Artificial Artificial Intelligence , 2011, AAAI.

[15]  Mausam,et al.  Dynamically Switching between Synergistic Workflows for Crowdsourcing , 2012, AAAI.

[16]  Krzysztof Z. Gajos,et al.  Toward automatic task design: a progress report , 2010, HCOMP '10.

[17]  Omar Alonso,et al.  Crowdsourcing for relevance evaluation , 2008, SIGF.

[18]  Peng Dai,et al.  Decision-Theoretic Control of Crowd-Sourced Workflows , 2010, AAAI.

[19]  John Darlington,et al.  A synthesis of several sorting algorithms , 1978, Acta Informatica.

[20]  David C. Parkes,et al.  Value-Based Policy Teaching with Active Indirect Elicitation , 2008, AAAI.

[21]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[22]  Eric Horvitz,et al.  Reasoning about beliefs and actions under computational resource constraints , 1987, Int. J. Approx. Reason..

[23]  Eric Horvitz,et al.  Combining human and machine intelligence in large-scale crowdsourcing , 2012, AAMAS.

[24]  Eric Joel Hovitz.  Computation and action under bounded resources , 1991 .