Predicting Game Difficulty and Engagement Using AI Players

This paper presents a novel approach to automated playtesting for the prediction of human player behavior and experience. We have previously demonstrated that Deep Reinforcement Learning (DRL) game-playing agents can predict both game difficulty and player engagement, operationalized as average pass and churn rates. We improve this approach by enhancing DRL with Monte Carlo Tree Search (MCTS). We also motivate an enhanced selection strategy for predictor features, based on the observation that an AI agent's best-case performance can yield stronger correlations with human data than the agent's average performance. Both additions consistently improve the prediction accuracy, and the DRL-enhanced MCTS outperforms both DRL and vanilla MCTS in the hardest levels. We conclude that player modelling via automated playtesting can benefit from combining DRL and MCTS. Moreover, it can be worthwhile to investigate a subset of repeated best AI agent runs, if AI gameplay does not yield good predictions on average.

[1]  Paul A. Cairns,et al.  Measuring perceived challenge in digital games: Development & validation of the challenge originating from recent gameplay interaction scale (CORGIS) , 2020, Int. J. Hum. Comput. Stud..

[2]  Karsten Rothmeier,et al.  Prediction of Player Churn and Disengagement Based on User Activity Data of a Freemium Online Strategy Game , 2020, IEEE Transactions on Games.

[3]  Xiaodong Li,et al.  Challenging AI: Evaluating the Effect of MCTS-Driven Dynamic Difficulty Adjustment on Player Enjoyment , 2019, ACSW.

[4]  Alex R. Wade,et al.  Modelling Early User-Game Interactions for Joint Estimation of Survival Time and Churn Probability , 2019, 2019 IEEE Conference on Games (CoG).

[5]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[6]  Julian Togelius,et al.  General Video Game Evaluation Using Relative Algorithm Performance Profiles , 2015, EvoApplications.

[7]  Elaine Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[8]  Elisa D. Mekler,et al.  Self-Determination Theory in HCI Games Research: Current Uses and Open Questions , 2020, CHI.

[9]  Perttu Hämäläinen,et al.  Predicting Mid-Air Interaction Movements and Fatigue Using Deep Reinforcement Learning , 2020, CHI.

[10]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[11]  Moataz Aly Kamaleldin Ahmed,et al.  Video Game Automated Testing Approaches: An Assessment Framework , 2020 .

[12]  Klaus Opwis,et al.  “An Odd Kind of Pleasure”: Differentiating Emotional Challenge in Digital Games , 2018, CHI.

[13]  Philippe Pasquier,et al.  A Generic Approach to Challenge Modeling for the Procedural Creation of Video Game Levels , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[14]  E. Deci,et al.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. , 2000, Contemporary educational psychology.

[15]  Paul A. Cairns,et al.  Predicting Player Experience without the Player.: An Exploratory Study , 2017, CHI PLAY.

[16]  Aysu Betin Can,et al.  Automated Video Game Testing Using Synthetic and Humanlike Agents , 2019, IEEE Transactions on Games.

[17]  James Alfred Walker,et al.  Metagame Autobalancing for Competitive Multiplayer Games , 2020, 2020 IEEE Conference on Games (CoG).

[18]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[19]  Antti Oulasvirta,et al.  It's time to rediscover HCI models , 2019, Interactions.

[20]  Aysu Betin Can,et al.  Enhancing the Monte Carlo Tree Search Algorithm for Video Game Testing , 2020, 2020 IEEE Conference on Games (CoG).

[21]  Julian Togelius,et al.  Automated Playtesting With Procedural Personas Through MCTS With Evolved Heuristics , 2018, IEEE Transactions on Games.

[22]  Paolo Burelli,et al.  Estimating Player Completion Rate in Mobile Puzzle Games Using Reinforcement Learning , 2020, 2020 IEEE Conference on Games (CoG).

[23]  Julian Togelius,et al.  An experiment in automatic game design , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[24]  Jesse Schell,et al.  The Art of Game Design: A book of lenses , 2019 .

[25]  Christian Guckelsberger,et al.  Challenge in Digital Games: Towards Developing a Measurement Tool , 2017, CHI Extended Abstracts.

[26]  Adam Finkelstein,et al.  Simulating strategy and dexterity for puzzle games , 2017, 2017 IEEE Conference on Computational Intelligence and Games (CIG).

[27]  Pejman Mirza-Babaei,et al.  Artificial Players in the Design Process: Developing an Automated Testing Tool for Game Level and World Design , 2020, CHI PLAY.

[28]  Samuel J. Gershman,et al.  Computational rationality: A converging paradigm for intelligence in brains, minds, and machines , 2015, Science.

[29]  Erik Ragnar Poromaa.  Crushing Candy Crush : Predicting Human Success Rate in a Mobile Game using Monte-Carlo Tree Search , 2017 .

[30]  Alexei A. Efros,et al.  Curiosity-Driven Exploration by Self-Supervised Prediction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[31]  Satinder Singh,et al.  Computational Rationality: Linking Mechanism and Behavior Through Bounded Utility Maximization , 2014, Top. Cogn. Sci..

[32]  Jenova Chen,et al.  Flow in games (and everything else) , 2007, CACM.

[33]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[34]  Adam M. Smith,et al.  Reveal-More: Amplifying Human Effort in Quality Assurance Testing Using Automated Exploration , 2019, 2019 IEEE Conference on Games (CoG).

[35]  Csaba Szepesvári,et al.  Bandit Based Monte-Carlo Planning , 2006, ECML.

[36]  Annie Louis,et al.  Beyond playing to win: Diversifying heuristics for GVGAI , 2017, 2017 IEEE Conference on Computational Intelligence and Games (CIG).

[37]  Georgios N. Yannakakis,et al.  TOWARDS OPTIMIZING ENTERTAINMENT IN COMPUTER GAMES , 2007, Appl. Artif. Intell..

[38]  Julian Togelius,et al.  Artificial Intelligence and Games , 2018, Springer International Publishing.

[39]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[40]  Konrad Tollmar,et al.  Using Deep Convolutional Neural Networks to Detect Rendered Glitches in Video Games , 2020, AIIDE.

[41]  Yuchul Shin,et al.  Playtesting in Match 3 Game Using Strategic Plays via Reinforcement Learning , 2020, IEEE Access.

[42]  Antonios Liapis,et al.  Dungeons & Replicants: Automated Game Balancing via Deep Player Behavior Modeling , 2020, 2020 IEEE Conference on Games (CoG).

[43]  Adam M. Smith,et al.  Monster Carlo: An MCTS-based Framework for Machine Playtesting Unity Games , 2018, 2018 IEEE Conference on Computational Intelligence and Games (CIG).

[44]  Maarten Löffler,et al.  Automated puzzle difficulty estimation , 2015, 2015 IEEE Conference on Computational Intelligence and Games (CIG).

[45]  Yan Zheng,et al.  Generating Behavior-Diverse Game AIs with Evolutionary Multi-Objective Deep Reinforcement Learning , 2020, IJCAI.

[46]  Peta Wyeth,et al.  GameFlow: a model for evaluating player enjoyment in games , 2005, CIE.

[47]  Jukka Corander,et al.  Inferring Cognitive Models from Data using Approximate Bayesian Computation , 2016, CHI.

[48]  David Reguera,et al.  The Physics of Fun: Quantifying Human Engagement into Playful Activities , 2019, 1911.01864.

[49]  Andrew K. Przybylski,et al.  The Motivational Pull of Video Games: A Self-Determination Theory Approach , 2006 .

[50]  R. Bellman.  A Markovian Decision Process , 1957 .

[51]  Stefan Freyr Gudmundsson,et al.  Human-Like Playtesting with Deep Learning , 2018, 2018 IEEE Conference on Computational Intelligence and Games (CIG).

[52]  Julian Togelius,et al.  General Video Game AI: Competition, Challenges and Opportunities , 2016, AAAI.

[53]  James M. Boyle,et al.  Engagement in digital entertainment games: A systematic review , 2012, Comput. Hum. Behav..

[54]  M. Csíkszentmihályi,et al.  The Concept of Flow , 2014 .

[55]  Paul A. Cairns,et al.  Not doing but thinking: the role of challenge in the gaming experience , 2012, CHI.

[56]  Jari Takatalo,et al.  Predicting Game Difficulty and Churn Without Players , 2020, CHI PLAY.

[57]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[58]  Joakim Bergdahl,et al.  Augmenting Automated Game Testing with Deep Reinforcement Learning , 2020, 2020 IEEE Conference on Games (CoG).

[59]  Julian Togelius,et al.  Experience-Driven Procedural Content Generation , 2011, IEEE Transactions on Affective Computing.

[60]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[61]  Paul A. Cairns,et al.  Emotional and Functional Challenge in Core and Avant-garde Games , 2015, CHI PLAY.

[62]  Kenneth R. Koedinger,et al.  Optimizing challenge in an educational game using large-scale design experiments , 2013, CHI.