Review of Intrinsic Motivation in Simulation-based Game Testing

This paper presents a review of intrinsic motivation in player modeling, with a focus on simulation-based game testing. Modern AI agents can learn to win many games; from a game testing perspective, a remaining research problem is how to model the aspects of human player behavior not explained by purely rational and goal-driven decision making. A major piece of this puzzle is constituted by intrinsic motivations, i.e., psychological needs that drive behavior without extrinsic reinforcement such as game score. We first review the common intrinsic motivations discussed in player psychology research and artificial intelligence, and then proceed to systematically review how the various motivations have been implemented in simulated player agents. Our work reveals that although motivations such as competence and curiosity have been studied in AI, work on utilizing them in simulation-based game testing is sparse, and other motivations such as social relatedness, immersion, and domination appear particularly underexplored.

[1]  R. Lazarus.  Progress on a cognitive-motivational-relational theory of emotion. , 1991, The American psychologist.

[2]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[3]  Annie Louis,et al.  Beyond playing to win: Diversifying heuristics for GVGAI , 2017, 2017 IEEE Conference on Computational Intelligence and Games (CIG).

[4]  P. White.  Appraisal Theory , 2015 .

[5]  Julian Togelius,et al.  Artificial Intelligence and Games , 2018, Springer International Publishing.

[6]  Andrew K. Przybylski,et al.  The Motivational Pull of Video Games: A Self-Determination Theory Approach , 2006 .

[7]  Filip De Turck,et al.  VIME: Variational Information Maximizing Exploration , 2016, NIPS.

[8]  Lennart E. Nacke,et al.  Biometric storyboards: visualising game user research data , 2012, CHI Extended Abstracts.

[9]  Cristina Conati,et al.  Modeling Motivation in a Social Network Game Using Player-Centric Traits and Personality Traits , 2015, UMAP.

[10]  Kyung-Joong Kim,et al.  MCTS with influence map for general video game playing , 2015, 2015 IEEE Conference on Computational Intelligence and Games (CIG).

[11]  Zahid Halim,et al.  Evolutionary Search in the Space of Rules for Creation of New Two-Player Board Games , 2014, Int. J. Artif. Intell. Tools.

[12]  Catholijn M. Jonker,et al.  Emotion in reinforcement learning agents and robots: a survey , 2017, Machine Learning.

[13]  Kathryn E. Merrick,et al.  Modeling motivation for adaptive nonplayer characters in dynamic computer game worlds , 2008, CIE.

[14]  Simon Colton,et al.  Intrinsically motivated general companion NPCs via Coupled Empowerment Maximisation , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[15]  Julian Togelius,et al.  Modifying MCTS for Human-Like General Video Game Playing , 2016, IJCAI.

[16]  Julian Togelius,et al.  Monte-Carlo tree search for persona based player modeling , 2015 .

[17]  K. Scherer,et al.  Appraisal processes in emotion: Theory, methods, research. , 2001 .

[18]  María Malfaz,et al.  A New Approach to Modeling Emotions and Their Use on a Decision-Making System for Artificial Agents , 2012, IEEE Transactions on Affective Computing.

[19]  Thomas W. Malone,et al.  Toward a Theory of Intrinsically Motivating Instruction , 1981, Cogn. Sci..

[20]  Daan Wierstra,et al.  Variational Intrinsic Control , 2016, ICLR.

[21]  Barbaros Bostan,et al.  A motivational framework for analyzing player and virtual agent behavior , 2010, Entertain. Comput..

[22]  Paul A. Cairns,et al.  Predicting Player Experience without the Player.: An Exploratory Study , 2017, CHI PLAY.

[23]  E. Deci,et al.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. , 2000, Contemporary educational psychology.

[24]  Shakir Mohamed,et al.  Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning , 2015, NIPS.

[25]  Marco Mirolli,et al.  Intrinsically Motivated Learning Systems: An Overview , 2013, Intrinsically Motivated Learning in Natural and Artificial Systems.

[26]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[27]  Yu-Hao Lee,et al.  Beyond player types: gaming achievement goal , 2011, SIGGRAPH '11.

[28]  Guillaume Chanel,et al.  A review of the use of psychophysiological methods in game research , 2011 .

[29]  Noah Wardrip-Fruin,et al.  Motivational game design patterns of 'ville games , 2012, FDG.

[30]  Julian Togelius,et al.  Generative agents for player decision modeling in games , 2014, FDG.

[31]  Karl M. Kapp,et al.  The Gamification of Learning and Instruction Fieldbook: Ideas into Practice , 2013 .

[32]  Chrystopher L. Nehaniv,et al.  General Self-Motivation and Strategy Identification: Case Studies Based on Sokoban and Pac-Man , 2014, IEEE Transactions on Computational Intelligence and AI in Games.

[33]  A. Elliot,et al.  A 2 X 2 achievement goal framework. , 2001, Journal of personality and social psychology.

[34]  Ben Goertzel,et al.  OpenPsi: A novel computational affective model and its application in video games , 2013, Engineering applications of artificial intelligence.

[35]  Christoph Salge,et al.  Empowerment - an Introduction , 2013, ArXiv.

[36]  Sam Devlin,et al.  Combining Gameplay Data with Monte Carlo Tree Search to Emulate Human Play , 2016, AIIDE.

[37]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[38]  Julian Togelius,et al.  General Video Game Level Generation , 2016, GECCO.

[39]  Pierre-Yves Oudeyer,et al.  How can we define intrinsic motivation , 2008 .

[40]  Kathryn E. Merrick,et al.  Motivated reinforcement learning for non-player characters in persistent computer game worlds , 2006, ACE '06.

[41]  J. C. Stanley.  Computer simulation of a model of habituation , 1976, Nature.

[42]  Vadim Bulitko,et al.  Maximizing Flow as a Metacontrol in Angband , 2021, AIIDE.

[43]  Nick Yee,et al.  Motivations for Play in Online Games , 2006, Cyberpsychology Behav. Soc. Netw..

[44]  Kenneth R. Koedinger,et al.  Is Difficulty Overrated?: The Effects of Choice, Novelty and Suspense on Intrinsic Motivation in Educational Games , 2017, CHI.

[45]  D. Dörner,et al.  PSI: A Computational Architecture of Cognition, Motivation, and Emotion , 2013 .

[46]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[47]  Jürgen Schmidhuber,et al.  A possibility for implementing curiosity and boredom in model-building neural controllers , 1991 .

[48]  Julian Togelius,et al.  Evolving personas for player decision modeling , 2014, 2014 IEEE Conference on Computational Intelligence and Games.

[49]  Ana Paiva,et al.  Emotion-Based Intrinsic Motivation for Reinforcement Learning Agents , 2011, ACII.

[50]  Mark O. Riedl,et al.  Monte-Carlo Tree Search for Simulation-based Strategy Analysis , 2019, ArXiv.

[51]  E. Deci,et al.  Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. , 2000, The American psychologist.

[52]  Nathan R Sturtevant,et al.  Deep Static and Dynamic Level Analysis: A Study on Infinite Mario , 2016 .

[53]  Marco Mirolli,et al.  Intrinsically Motivated Learning in Natural and Artificial Systems , 2013 .

[54]  Tom Schaul,et al.  Unifying Count-Based Exploration and Intrinsic Motivation , 2016, NIPS.

[55]  Michael Katchabaw,et al.  Enabling motivated believable agents with reinforcement learning , 2014, 2014 IEEE Games Media Entertainment.

[56]  Julian Togelius,et al.  Evolving Game Skill-Depth using General Video Game AI agents , 2017, 2017 IEEE Congress on Evolutionary Computation (CEC).

[57]  Julian Togelius,et al.  Imitating human playing styles in Super Mario Bros , 2013, Entertain. Comput..

[58]  Charles Butler.  Applied Behavioral Economics: A Game Designer’s Perspective , 2015 .

[59]  Nuttapong Chentanez,et al.  Intrinsically Motivated Reinforcement Learning , 2004, NIPS.

[60]  Filip De Turck,et al.  Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural Networks , 2016, ArXiv.

[61]  Kathryn E. Merrick,et al.  Achievement, affiliation, and power: Motive profiles for artificial agents , 2011, Adapt. Behav..

[62]  Xiaodong Li,et al.  Learning a Super Mario controller from examples of human play , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[63]  S. Reiss,et al.  Toward a comprehensive assessment of fundamental motivation : Factor structure of the reiss profiles , 1998 .

[64]  Andrew G. Barto,et al.  Intrinsic Motivation and Reinforcement Learning , 2013, Intrinsically Motivated Learning in Natural and Artificial Systems.

[65]  Jürgen Schmidhuber,et al.  Optimal Artificial Curiosity, Creativity, Music, and the Fine Arts , 2005 .

[66]  Ilya Kostrikov,et al.  Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play , 2017, ICLR.

[67]  Daniel Polani,et al.  Control Capacity of Partially Observable Dynamic Systems in Continuous Time , 2017, ArXiv.

[68]  Chrystopher L. Nehaniv,et al.  Empowerment: a universal agent-centric measure of control , 2005, 2005 IEEE Congress on Evolutionary Computation.

[69]  Regan L. Mandryk,et al.  BrainHex: A neurobiological gamer typology survey , 2014, Entertain. Comput..

[70]  Kennon M. Sheldon,et al.  What is satisfying about satisfying events? Testing 10 candidate psychological needs. , 2001, Journal of personality and social psychology.

[71]  Andrew Nealen,et al.  Exploring Game Space Using Survival Analysis , 2015, FDG.

[72]  K. Scherer.  Appraisal considered as a process of multilevel sequential checking. , 2001 .

[73]  A. F. Adams,et al.  The Survey , 2021, Dyslexia in Higher Education.

[74]  John E. Laird,et al.  Emotion-Driven Reinforcement Learning , 2008 .

[75]  P. Silvia.  Interest—The Curious Emotion , 2008 .

[76]  Lennart E. Nacke,et al.  The kaleidoscope of effective gamification: deconstructing gamification in business applications , 2013, Gamification.

[77]  Julian Togelius,et al.  Experience-Driven Procedural Content Generation , 2011, IEEE Transactions on Affective Computing.

[78]  Joeri Hofmans,et al.  Relating Specific Emotions to Intrinsic Motivation: On the Moderating Role of Positive and Negative Emotion Differentiation , 2014, PloS one.

[79]  Ben Goertzel,et al.  A world survey of artificial brain projects, Part II: Biologically inspired cognitive architectures , 2010, Neurocomputing.

[80]  Kathryn E. Merrick,et al.  The Role of Implicit Motives in Strategic Decision-Making: Computational Models of Motivated Learning and the Evolution of Motivated Agents , 2015, Games.

[81]  N. Lazzaro.  Why we Play Games: Four Keys to More Emotion without Story , 2004 .

[82]  Honglak Lee,et al.  Action-Conditional Video Prediction using Deep Networks in Atari Games , 2015, NIPS.

[83]  Jürgen Schmidhuber,et al.  Curious model-building control systems , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[84]  Kokolo Ikeda,et al.  Learning human-like behaviors using neuroevolution with statistical penalties , 2017, 2017 IEEE Conference on Computational Intelligence and Games (CIG).

[85]  Daniele Loiacono,et al.  Player Modeling , 2013, Artificial and Computational Intelligence in Games.

[86]  S. Shankar Sastry,et al.  Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning , 2017, ArXiv.

[87]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[88]  Alexei A. Efros,et al.  Curiosity-Driven Exploration by Self-Supervised Prediction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[89]  Sergey Levine,et al.  Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models , 2015, ArXiv.

[90]  Matthew K. Miller,et al.  How Self-Esteem Shapes our Interactions with Play Technologies , 2015, CHI PLAY.

[91]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[92]  Juho Hamari,et al.  Player Types: A Meta-synthesis , 2014, Trans. Digit. Games Res. Assoc..

[93]  Mark O. Riedl,et al.  Monte-Carlo Tree Search for Simulation-based Play Strategy Analysis , 2015, FDG.

[94]  Zhao Chen,et al.  The Game Imitation: Deep Supervised Convolutional Networks for Quick Video Game AI , 2017, ArXiv.

[95]  H. Murray,et al.  Explorations in Personality , 2007 .

[96]  S. Reiss.  Multifaceted Nature of Intrinsic Motivation: The Theory of 16 Basic Desires , 2004 .