Generalized Task Markets for Human and Machine Computation

We discuss challenges and opportunities for developing generalized task markets where human and machine intelligence are enlisted to solve problems, based on a consideration of the competencies, availabilities, and pricing of different problem-solving resources. The approach couples human computation with machine learning and planning, and is aimed at optimizing the flow of subtasks to people and to computational problem solvers. We illustrate key ideas in the context of Lingua Mechanica, a project focused on harnessing human and machine translation skills to perform translation among languages. We present infrastructure and methods for enlisting and guiding human and machine computation for language translation, including details about the hardness of generating plans for assigning tasks to solvers. Finally, we discuss studies performed with machine and human solvers, focusing on components of a Lingua Mechanica prototype.

[1]  Desney S. Tan,et al.  Complementary computing for visual tasks: Meshing computer vision with human visual processing , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[2]  Jan Vondrák,et al.  Optimal approximation for the submodular welfare problem in the value oracle model , 2008, STOC.

[3]  Luis von Ahn.  Human Computation , 2008, ICDE.

[4]  Uriel Feige,et al.  On maximizing welfare when utility functions are subadditive , 2006, STOC '06.

[5]  Chris Quirk,et al.  Discriminative, Syntactic Language Modeling through Latent SVMs , 2008 .

[6]  Eric Horvitz,et al.  Reflections on Challenges and Promises of Mixed-Initiative Interaction , 2007, AI Mag..

[7]  Chris Callison-Burch,et al.  Improving statistical translation through editing , 2004, EAMT.

[8]  Sarit Kraus,et al.  Task Allocation Via Coalition Formation Among Autonomous Agents , 1995, IJCAI.

[9]  Vahab S. Mirrokni,et al.  Tight information-theoretic lower bounds for welfare maximization in combinatorial auctions , 2008, EC '08.

[10]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[11]  Eric Horvitz,et al.  Complementary computing: policies for transferring callers from dialog systems to human receptionists , 2006, User Modeling and User-Adapted Interaction.

[12]  Dafna Shahaf,et al.  Investigation of Human-Computer Task Markets: Methods and Prototype , 2009 .

[13]  M. L. Fisher,et al.  An analysis of approximations for maximizing submodular set functions—I , 1978, Math. Program..