Combining crowdsourcing and learning to improve engagement and performance

Crowdsourcing complex creative tasks remains difficult, in part because these tasks require skilled workers. Most crowdsourcing platforms do not help workers acquire the skills necessary to accomplish complex creative tasks. In this paper, we describe a platform that combines learning and crowdsourcing to benefit both the workers and the requesters. Workers gain new skills through interactive step-by-step tutorials and test their knowledge by improving real-world images submitted by requesters. In a series of three deployments spanning two years, we varied the design of our platform to enhance the learning experience and improve the quality of the crowd work. We tested our approach in the context of LevelUp for Photoshop, which teaches people how to do basic photograph improvement tasks using Adobe Photoshop. We found that by using our system workers gained new skills and produced high-quality edits for requested images, even if they had little prior experience editing images.

[1]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[2]  J. Guilford,et al.  The nature of human intelligence. , 1968 .

[3]  Teresa M. Amabile,et al.  Creativity and Innovation in Organizations , 1996 .

[4]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[5]  Tessa A. Lau,et al.  DocWizards: a system for authoring follow-me documentation wizards , 2005, UIST.

[6]  John Baer,et al.  Essentials of Creativity Assessment , 2008 .

[7]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[8]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[9]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[10]  Michael S. Bernstein,et al.  Crowds in two seconds: enabling realtime crowd-powered interfaces , 2011, UIST.

[11]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[12]  Elizabeth Gerber,et al.  Affective computational priming and creativity , 2011, CHI.

[13]  Mira Dontcheva,et al.  Pause-and-play: automatically linking screencast video tutorials with applications , 2011, UIST.

[14]  Tovi Grossman,et al.  GamiCAD: a gamified tutorial system for first time autocad users , 2012, UIST.

[15]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[16]  Björn Hartmann,et al.  Collaboratively crowdsourcing workflows with turkomatic , 2012, CSCW.

[17]  Tao Dong,et al.  Discovery-based games for learning software , 2012, CHI.

[18]  Elizabeth Gerber,et al.  Affect and Creative Performance on Crowdsourcing Platforms , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[19]  Aniket Kittur,et al.  Reviewing versus doing: learning and performance in crowd assessment , 2014, CSCW.