Examining the Effectiveness of Gamification in Human Computation

ABSTRACT Within the human computation paradigm, gamification is increasingly gaining interest. This is because an enjoyable experience generated by game features can be a powerful approach to attract participants. Although potentially useful, little research has been conducted into understanding the effectiveness of gamification in human computation. In this experimental study, we operationalized effectiveness as perceived engagement and user acceptance and examined it by comparing the performance of a gamified human computation system against a non-gamified version. We also investigate the determinants of acceptance and how their effects differ between these two systems. Analysis of our data found that participants experienced more engagement and showed higher behavioral intentions toward the gamified system. Moreover, perceived output quality and perceived engagement were significant determinants of acceptance of the gamified system. In contrast, determinants for acceptance of the non-gamified system were perceived output quality and perceived usability.

[1]  E. Deci,et al.  The general causality orientations scale: Self-determination in personality , 1985 .

[2]  R. Haase,et al.  Multivariate analysis of variance. , 1987 .

[3]  Richard N. Landers,et al.  Gamification of task performance with leaderboards: A goal setting experiment , 2017, Comput. Hum. Behav..

[4]  Andrew K. Przybylski,et al.  The Motivational Pull of Video Games: A Self-Determination Theory Approach , 2006 .

[5]  Luis de Marcos,et al.  An empirical study comparing gamification and social networking on e-learning , 2014, Comput. Educ..

[6]  Chei Sian Lee,et al.  Perceptions, quality and motivational needs in image tagging human computation games , 2011, J. Inf. Sci..

[7]  Juho Hamari,et al.  Why Do People Play Games? A Review of Studies on Adoption and Use , 2015, 2015 48th Hawaii International Conference on System Sciences.

[8]  Sung S. Kim,et al.  Out of Dedication or Constraint? A Dual Model of Post-Adoption Phenomena and its Empirical Test in the Context of Online Services , 2009, MIS Q..

[9]  Chei Sian Lee,et al.  "I played games as there was nothing else to do": Understanding motivations for using mobile content sharing games , 2012, Online Inf. Rev..

[10]  Peta Wyeth,et al.  GameFlow: a model for evaluating player enjoyment in games , 2005, CIE.

[11]  Marina Krcmar,et al.  Conceptualizing Media Enjoyment as Attitude: Implications for Mass Media Effects Research , 2004 .

[12]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[13]  Michael G. Morris,et al.  User Acceptance of Information Technology: Theories and Models , 1996 .

[14]  N. Sadat Shami,et al.  Experiments on Motivational Feedback for Crowdsourced Workers , 2013, ICWSM.

[15]  Jeanne H. Brockmyer,et al.  The Development of the Game Engagement Questionnaire: A Measure of Engagement in Video Game Playing: Response to Reviews , 2009, Interacting with computers.

[16]  Jakub Simko,et al.  Human computation: Image metadata acquisition based on a single-player annotation game , 2013, Int. J. Hum. Comput. Stud..

[17]  Jean Écalle,et al.  Serious games as new educational tools: how effective are they? A meta-analysis of recent studies , 2013, J. Comput. Assist. Learn..

[18]  Ethan Mollick,et al.  Mandatory Fun: Consent, Gamification and the Impact of Games at Work , 2014 .

[19]  Chei Sian Lee,et al.  A typology of human computation games: an analysis and a review of current games , 2015, Behav. Inf. Technol..

[20]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[21]  James R. Lewis,et al.  Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies , 2002, Int. J. Hum. Comput. Interact..

[22]  Premkumar T. Devanbu,et al.  How social Q&A sites are changing knowledge sharing in open source software communities , 2014, CSCW.

[23]  Remco C. Veltkamp,et al.  Studying emotion induced by music through a crowdsourcing game , 2016, Inf. Process. Manag..

[24]  Juho Hamari,et al.  Demographic differences in perceived benefits from gamification , 2014, Comput. Hum. Behav..

[25]  Laura A. Dabbish,et al.  Designing games with a purpose , 2008, CACM.

[26]  Ee-Peng Lim,et al.  Understanding the determinants of human computation game acceptance: The effects of aesthetic experience and output quality , 2016, Online Inf. Rev..

[27]  Juho Hamari,et al.  Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification , 2014, 2014 47th Hawaii International Conference on System Sciences.

[28]  I. Ajzen,et al.  Attitude-behavior relations: A theoretical analysis and review of empirical research. , 1977 .

[29]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[30]  Ingoo Han,et al.  The impact of Web quality and playfulness on user acceptance of online retailing , 2007, Inf. Manag..

[31]  Alessandro Canossa,et al.  Gamifying Research: Strategies, Opportunities, Challenges, Ethics , 2015, CHI Extended Abstracts.

[32]  Pearl Brereton,et al.  Does the technology acceptance model predict actual use? A systematic literature review , 2010, Inf. Softw. Technol..

[33]  Sung-Joon Yoon,et al.  Predicting the use of online information services based on a modified UTAUT model , 2014, Behav. Inf. Technol..

[34]  Juho Hamari,et al.  "Working out for likes": An empirical study on social influence in exercise gamification , 2015, Comput. Hum. Behav..

[35]  Fred D. Davis.  User Acceptance of Information Technology: System Characteristics, User Perceptions and Behavioral Impacts , 1993, Int. J. Man Mach. Stud..

[36]  Matthew K. O. Lee,et al.  Acceptance of Internet-based learning medium: the role of extrinsic and intrinsic motivation , 2005, Inf. Manag..

[37]  Robert Davis,et al.  Modeling game usage, purchase behavior and ease of use , 2012, Entertain. Comput..

[38]  Martin Schader,et al.  Personalized task recommendation in crowdsourcing information systems - Current state of the art , 2014, Decis. Support Syst..

[39]  Alton Yeow-Kuan Chua,et al.  Indagator: Investigating perceived gratifications of an application that blends mobile content sharing with gameplay , 2010, J. Assoc. Inf. Sci. Technol..

[40]  Alton Yeow-Kuan Chua,et al.  Fight or unite: Investigating game genres for image tagging , 2011, J. Assoc. Inf. Sci. Technol..

[41]  Brandon G. Morton,et al.  Improving music emotion labeling using human computation , 2010, HCOMP '10.

[42]  Juho Hamari,et al.  Gamification in Crowdsourcing: A Review , 2016, 2016 49th Hawaii International Conference on System Sciences (HICSS).

[43]  Tzung-Ru Tsai,et al.  What Drives People to Continue to Play Online Games? An Extension of Technology Model and Theory of Planned Behavior , 2010, Int. J. Hum. Comput. Interact..

[44]  M. Sherif,et al.  The psychology of attitudes. , 1946, Psychological review.

[45]  Alexandre N. Tuch,et al.  Disassembling gamification: the effects of points and meaning on user motivation and performance , 2013, CHI Extended Abstracts.

[46]  Lennart E. Nacke,et al.  From game design elements to gamefulness: defining "gamification" , 2011, MindTrek.

[47]  Irwin King,et al.  A Survey of Human Computation Systems , 2009, 2009 International Conference on Computational Science and Engineering.

[48]  James M. Boyle,et al.  A systematic literature review of empirical evidence on computer games and serious games , 2012, Comput. Educ..

[49]  Luis von Ahn.  Human Computation , 2008, ICDE.

[50]  Elaine Toms,et al.  Examining the generalizability of the User Engagement Scale (UES) in exploratory search , 2013, Inf. Process. Manag..

[51]  Juho Hamari,et al.  Do badges increase user activity? A field experiment on the effects of gamification , 2017, Comput. Hum. Behav..

[52]  Deborah I Thompson,et al.  Playing for real: video games and stories for health-related behavior change. , 2008, American journal of preventive medicine.

[53]  François Bry,et al.  Human computation , 2018, it Inf. Technol..

[54]  Eric N. Wiebe,et al.  Measuring engagement in video game-based environments: Investigation of the User Engagement Scale , 2014, Comput. Hum. Behav..

[55]  Juho Hamari,et al.  A definition for gamification: anchoring gamification in the service marketing literature , 2016, Electronic Markets.

[56]  Andreas Sonderegger,et al.  The influence of design aesthetics in usability testing: effects on user performance and perceived usability. , 2010, Applied ergonomics.

[57]  Carl Gutwin,et al.  Effects of gamification on participation and data quality in a real-world market research domain , 2013, Gamification.