Crowd vs. crowd: large-scale cooperative design through open team competition

Following the recent remarkable successes of crowdsourcing, there have been attempts to apply it to design. However a design problem is often too complex and difficult to break down into simpler, distributable tasks as required by the conventional crowdsourcing model. In this paper, we present Crowd vs. Crowd (CvC), a novel design crowdsourcing method, where several design teams made up of designers and crowd compete with each other. In each team, a designer coordinates effective communication between the crowd members and takes responsibility for the final design output, and the crowd contributes at different stages of design. We conducted an initial evaluation of CvC in comparison with other collaborative design methods, and found that: CvC can attract more people to participate; the crowd can make useful contribution in CvC; CvC can produce competent design outputs. We then applied CvC to two real-life design problems: first, designing a new logo for a university department; second, for a small tech company. With quantitative and qualitative analyses on these applications, we observed that the elements of competition and collaboration helped to sustain the crowd's motivation to participate, and to produce quality design outcomes with higher level of satisfaction for the stakeholders.

[1]  Finn Kensing,et al.  Participatory Design: Issues and Concerns , 2004, Computer Supported Cooperative Work (CSCW).

[2]  Uta Brandes,et al.  Design by use : the everyday metamorphosis of things , 2008 .

[3]  Karim R. Lakhani,et al.  Open Innovation and Organizational Boundaries: The Impact of Task Decomposition and Knowledge Distribution on the Locus of Innovation , 2012 .

[4]  Aniket Kittur,et al.  The polymath project: lessons from a successful online collaboration in mathematics , 2011, CHI.

[5]  Amy Bruckman,et al.  Leadership in online creative collaboration , 2008, CSCW.

[6]  Aniket Kittur,et al.  Crowdsourcing, collaboration and creativity , 2010, XRDS.

[7]  E. Deci,et al.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. , 2000, Contemporary educational psychology.

[8]  Karim R. Lakhani,et al.  Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis , 2011, Manag. Sci..

[9]  Brian P. Bailey,et al.  What do you think?: a case study of benefit, expectation, and interaction in a large online critique community , 2012, CSCW.

[10]  Greg Little,et al.  TurKit: Tools for iterative tasks on mechanical turk , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[11]  Bill Buxton,et al.  Sketching User Experiences: Getting the Design Right and the Right Design , 2007 .

[12]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[13]  Kathrin M. Möslein,et al.  Overcoming Mass Confusion: Collaborative Customer Co-Design in Online Communities , 2005, J. Comput. Mediat. Commun..

[14]  Aniket Kittur,et al.  CrowdWeaver: visually managing complex crowd work , 2012, CSCW.

[15]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[16]  Sebastian Boring,et al.  #EpicPlay: crowd-sourcing sports video highlights , 2012, CHI.

[17]  Peter Norvig,et al.  Can Distributed Volunteers Accomplish Massive Data Analysis Tasks , 2001 .

[18]  Matthew J. Salganik,et al.  Wiki surveys : Open and quantifiable social data collection ∗ , 2012 .

[19]  Joseph M. Reagle,et al.  Do as I do:: authorial leadership in wikipedia , 2007, WikiSym '07.

[20]  Brian L. Sullivan,et al.  eBird: A citizen-based bird observation network in the biological sciences , 2009 .

[21]  Lydia B. Chilton,et al.  TurKit: Tools for iterative tasks on mechanical turk , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[22]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[23]  Jeffrey V. Nickerson,et al.  Cooks or cobblers?: crowd creativity through combination , 2011, CHI.

[24]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[25]  Björn Hartmann,et al.  Collaboratively crowdsourcing workflows with turkomatic , 2012, CSCW.

[26]  Björn Hartmann,et al.  CommunitySourcing: engaging local crowds to perform expert work via physical kiosks , 2012, CHI.