Reviewing versus doing: learning and performance in crowd assessment

In modern crowdsourcing markets, requesters face the challenge of training and managing large transient workforces. Requesters can hire peer workers to review others' work, but the value may be marginal, especially if the reviewers lack requisite knowledge. Our research explores if and how workers learn and improve their performance in a task domain by serving as peer reviewers. Further, we investigate whether peer reviewing may be more effective in teams where the reviewers can reach consensus through discussion. An online between-subjects experiment compares the trade-offs of reviewing versus producing work using three different organization strategies: working individually, working as an interactive team, and aggregating individuals into nominal groups. The results show that workers who review others' work perform better on subsequent tasks than workers who just produce. We also find that interactive reviewer teams outperform individual reviewers on all quality measures. However, aggregating individual reviewers into nominal groups produces better quality assessments than interactive teams, except in task domains where discussion helps overcome individual misconceptions.

[1]  Daniel L. Schwartz,et al.  Teachable Agents and the Protégé Effect: Increasing the Effort Towards Learning , 2009 .

[2]  P. Resnick,et al.  Building Successful Online Communities: Evidence-Based Social Design , 2012 .

[3]  Björn Hartmann,et al.  MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture , 2012, IEEE Internet Computing.

[4]  J. Frederiksen,et al.  Inquiry, Modeling, and Metacognition: Making Science Accessible to All Students , 1998 .

[5]  I. Sigel,et al.  HANDBOOK OF CHILD PSYCHOLOGY , 2006 .

[6]  David Barry,et al.  Managing the bossless team: Lessons in distributed leadership , 1991 .

[7]  Terence R. Mitchell,et al.  How, When, and Why Bad Apples Spoil the Barrel: Negative Group Members and Dysfunctional Groups , 2006 .

[8]  I. Janis.  Groupthink: Psychological Studies of Policy Decisions and Fiascoes , 1982 .

[9]  G. Yukl,et al.  Leadership in Organizations , 1981 .

[10]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[11]  Aniket Kittur,et al.  Identifying shared leadership in Wikipedia , 2011, CHI.

[12]  J. Guilford,et al.  The nature of human intelligence. , 1968 .

[13]  Aniket Kittur,et al.  Organizing without formal organization: group identification, goal setting and social modeling in directing online production , 2012, CSCW.

[14]  A. Delbecq,et al.  Nominal Versus Interacting Group Processes for Committee Decision-Making Effectiveness , 1971 .

[15]  E.,et al.  GROUPS : INTERACTION AND PERFORMANCE , 2001 .

[16]  N. Kerr,et al.  Group performance and decision making. , 2004, Annual review of psychology.

[17]  Patrick R. Laughlin,et al.  Collective induction: Mutual group and individual influence by exchange of hypotheses and evidence , 1986 .

[18]  Ann L. Brown.  Learning, remembering, and understanding , 1982 .

[19]  K. M. Bartol,et al.  Empowering Leadership in Management Teams: Effects on Knowledge Sharing, Efficacy, And Performance , 2006 .

[20]  Mary T. Dzindolet,et al.  Perception of Performance in Group Brainstorming: The Illusion of Group Productivity , 1993 .

[21]  H. P. Sims,et al.  Business Without Bosses: How Self-Managing Teams Are Building High- Performing Companies , 1993 .

[22]  M. Neale,et al.  What Differences Make a Difference? , 2005, Psychological science in the public interest : a journal of the American Psychological Society.

[23]  D. Meyer,et al.  Supporting Online Material Materials and Methods Som Text Figs. S1 to S6 References Evidence for a Collective Intelligence Factor in the Performance of Human Groups , 2022 .

[24]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[25]  M. Merry.  Shared Leadership: Reframing the Hows and Whys of Leadership , 2004 .

[26]  M. Diehl,et al.  Why Groups are less Effective than their Members: On Productivity Losses in Idea-generating Groups , 1994 .

[27]  Evelyn Yarzebinski,et al.  Studying the Effect of Tutor Learning Using a Teachable Agent that Asks the Student Tutor for Explanations , 2012, 2012 IEEE Fourth International Conference On Digital Game And Intelligent Toy Enhanced Learning.

[28]  A. V. D. Ven,et al.  The Effectiveness of Nominal, Delphi, and Interacting Group Decision Making Processes , 1974 .

[29]  E. Salas,et al.  Productivity loss in brainstorming groups: A meta-analytic integration. , 1991 .

[30]  D. Isenberg.  Group polarization: A critical review and meta-analysis. , 1986 .

[31]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[32]  E. Deci,et al.  Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. , 2000, The American psychologist.

[33]  H Gu,et al.  The effects of averaging subjective probability estimates between and within judges. , 2000, Journal of experimental psychology. Applied.

[34]  Andrew B. Hargadon,et al.  Brainstorming groups in context: Effectiveness in a product design firm , 1996 .

[35]  Aniket Kittur,et al.  Harnessing the wisdom of crowds in wikipedia: quality through coordination , 2008, CSCW.

[36]  Daniel Kahneman,et al.  Deliberating About Dollars: The Severity Shift , 2000 .

[37]  J. Levine,et al.  Progress in Small Group Research , 1990 .

[38]  Aniket Kittur,et al.  Effectiveness of shared leadership in online communities , 2012, CSCW.

[39]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[40]  John Joseph Horton,et al.  Employer Expectations, Peer Effects and Productivity: Evidence from a Series of Field Experiments , 2010, ArXiv.