Gamification in Crowdsourcing: A Review

This study investigates how different gamification implementations can increase crowdsourcees' motivation and participation in crowdsourcing (CS). To this end, we review empirical literature that has investigated the use of gamification in crowdsourcing settings. Overall, the results of the review indicate that gamification has been an effective approach for increasing crowdsourcing participation. When comparing crowdcreating, -solving, -processing and-rating CS approaches, the results show differences in the use of gamification across CS types. Crowdsourcing initiatives that provide more monotonous tasks most commonly used mere points and other simpler gamification implementations, whereas CS initiatives that seek for diverse and creative contributions have employed gamification in more manifold ways employing a richer set of mechanics. These findings provide insights for designers of gamified systems and further research on the topics of gamification and crowdsourcing.

[1]  Shin Saito,et al.  Introducing game elements in crowdsourced video captioning by non-experts , 2014, W4A.

[2]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[3]  David Coyle,et al.  Competing or aiming to be average?: normification as a means of engaging digital volunteers , 2014, CSCW.

[4]  Steffen P. Walz,et al.  Gamification and Economics , 2014 .

[5]  Christof Weinhardt,et al.  How (not) to Incent Crowd Workers , 2015, Bus. Inf. Syst. Eng..

[6]  Pratik Mishra,et al.  Incentive design to mould online behavior: a game mechanics perspective , 2014, GamifIR '14.

[7]  Joo-Ho Lee,et al.  Cooperative robot software development through the internet , 2014, 2014 IEEE/SICE International Symposium on System Integration.

[8]  A. Pentland,et al.  Collective intelligence , 2006, IEEE Comput. Intell. Mag..

[9]  Marc Jay,et al.  Using crowdsourcing to support pro-environmental community activism , 2013, CHI.

[10]  Padmini Srinivasan,et al.  Quality through flow and immersion: gamifying crowdsourced relevance assessments , 2012, SIGIR '12.

[11]  Juho Hamari,et al.  Defining gamification: a service marketing perspective , 2012, MindTrek.

[12]  Jon Chamberlain The annotation-validation (AV) model: rewarding contribution using retrospective agreement , 2014, GamifIR '14.

[13]  Premkumar T. Devanbu,et al.  How social Q&A sites are changing knowledge sharing in open source software communities , 2014, CSCW.

[14]  Tatsuo Nakajima,et al.  Mobile Image Search via Local Crowd: A User Study , 2011, 2011 IEEE 17th International Conference on Embedded and Real-Time Computing Systems and Applications.

[15]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[16]  E. Deci Effects of Externally Mediated Rewards on Intrinsic Motivation. , 1971 .

[17]  C. Lintott,et al.  Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey , 2008, 0804.4483.

[18]  E. Deci,et al.  A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. , 1999, Psychological bulletin.

[19]  Juho Hamari,et al.  Transforming homo economicus into homo ludens: A field experiment on gamification in a utilitarian peer-to-peer trading service , 2013, Electron. Commer. Res. Appl..

[20]  Hironobu Takagi,et al.  Involving Senior Workers in Crowdsourced Proofreading , 2014, HCI.

[21]  Axel Küpper,et al.  Turning the OpenMobileNetwork into a live crowdsourcing platform for semantic context-aware services , 2013, I-SEMANTICS '13.

[22]  Masamichi Shimosaka,et al.  Steered crowdsensing: incentive design towards quality-oriented place-centric crowdsensing , 2014, UbiComp.

[23]  Leif Azzopardi,et al.  Studying user browsing behavior through gamified search tasks , 2014, GamifIR '14.

[24]  Joseph S. Valacich,et al.  Enhancing the Motivational Affordance of Information Systems: The Effects of Real-Time Performance Feedback and Goal Setting in Group Collaboration Environments , 2010, Manag. Sci..

[25]  Qinghua Zhu,et al.  Evaluation on crowdsourcing research: Current status and future direction , 2012, Information Systems Frontiers.

[26]  Tatsuo Nakajima,et al.  Gamifying intelligent environments , 2011, Ubi-MUI '11.

[27]  François Bry,et al.  Human computation , 2018, it Inf. Technol..

[28]  Juho Hamari,et al.  Demographic differences in perceived benefits from gamification , 2014, Comput. Hum. Behav..

[29]  Christopher G. Harris The beauty contest revisited: measuring consensus rankings of relevance using a game , 2014, GamifIR '14.

[30]  Peter Norvig,et al.  Can Distributed Volunteers Accomplish Massive Data Analysis Tasks , 2001 .

[31]  Heejin Choi,et al.  A Study about Designing Reward for Gamified Crowdsourcing System , 2014, HCI.

[32]  Lee Yik Sheng,et al.  Modelling learning from Ingress (Google's augmented reality social game) , 2013, 2013 IEEE 63rd Annual Conference International Council for Education Media (ICEM).

[33]  Woonhee Sung,et al.  GREENIFY: A Real-World Action Game for Climate Change Education , 2013 .

[34]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[35]  N. Sadat Shami,et al.  Experiments on Motivational Feedback for Crowdsourced Workers , 2013, ICWSM.

[36]  Ebroul Izquierdo,et al.  People recognition using gamified ambiguous feedback , 2014, GamifIR '14.

[37]  Paul D. Ellis,et al.  The essential guide to effect sizes : statistical power, meta-analysis, and the interpretation of research results , 2010 .

[38]  Michitaka Hirose,et al.  T-echo: Promoting Intergenerational Communication through Gamified Social Mentoring , 2014, HCI.

[39]  Juho Hamari,et al.  Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification , 2014, 2014 47th Hawaii International Conference on System Sciences.

[40]  Paul J. Krause,et al.  Tiger Nation: Empowering citizen scientists , 2012, 2012 6th IEEE International Conference on Digital Ecosystems and Technologies (DEST).

[41]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[42]  Reiko Hishiyama,et al.  Analysis of self-tagging during conversational chat in multilingual gaming simulation , 2013, Second International Conference on Future Generation Communication Technologies (FGCT 2013).

[43]  Juho Hamari,et al.  The sharing economy: Why people participate in collaborative consumption , 2016, J. Assoc. Inf. Sci. Technol..

[44]  Jan Marco Leimeister,et al.  Collective Intelligence , 2010, Bus. Inf. Syst. Eng..

[45]  Qinghua Zhu,et al.  Effects of extrinsic and intrinsic motivation on participation in crowdsourcing contest: A perspective of self-determination theory , 2014, Online Inf. Rev..

[46]  Tatsuo Nakajima,et al.  Gamifying Social Media to Encourage Social Activities with Digital-Physical Hybrid Role-Playing , 2014, HCI.

[47]  Juho Hamari,et al.  International Journal of Information Management Why Do People Use Gamification Services? , 2022 .

[48]  Jennifer Preece,et al.  Using gamification to inspire new citizen science volunteers , 2013, Gamification.

[49]  Jorge Gonçalves,et al.  Game of words: tagging places through crowdsourcing on public displays , 2014, Conference on Designing Interactive Systems.

[50]  Lora Aroyo,et al.  Dr. Detective: combining gamification techniques and crowdsourcing to create a gold standard for the medical domain , 2013 .

[51]  Panagiotis G. Ipeirotis,et al.  Quizz: targeted crowdsourcing with a billion (potential) users , 2014, WWW.

[52]  Martin Schader,et al.  Personalized task recommendation in crowdsourcing information systems - Current state of the art , 2014, Decis. Support Syst..

[53]  Dahui Li,et al.  Task Design, Motivation, and Participation in Crowdsourcing Contests , 2011, Int. J. Electron. Commer..

[54]  Hironobu Takagi,et al.  Skill Development Framework for Micro-Tasking , 2014, HCI.