Reveal-More: Amplifying Human Effort in Quality Assurance Testing Using Automated Exploration

Attempting to maximize coverage of a game via human gameplay is laborious and repetitive, introducing delays in the development process. Despite the importance of quality assurance (QA) testing, QA remains an underinvested area in the technical games research community. In this paper, we show that relatively simple automatic exploration techniques can be used to multiplicatively amplify coverage of a game starting from human tester data. Instead of attempting to displace human QA efforts, we seek to grow the impact that a human tester can make. Experiments with two games for the Super Nintendo Entertainment System highlight the qualitative and quantitative differences between isolated human and machine play compared to our hybrid approach called Reveal-More. We contribute a QA testing workflow that scales with the amount of human and machine time allocated to the effort.

[1]  Alexei A. Efros,et al.  Large-Scale Study of Curiosity-Driven Learning , 2018, ICLR.

[2]  Gang Xiao,et al.  Software Testing by Active Learning for Commercial Games , 2005, AAAI.

[3]  Adam M. Smith,et al.  Taking the Scenic Route: Automatic Exploration for Videogames , 2019, KEG@AAAI.

[4]  Tom Schaul,et al.  A video game description language for model-based or interactive learning , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[5]  Julian Togelius,et al.  General Video Game AI: A Multitrack Framework for Evaluating Agents, Games, and Content Generation Algorithms , 2018, IEEE Transactions on Games.

[6]  Julian Togelius,et al.  Evolving personas for player decision modeling , 2014, 2014 IEEE Conference on Computational Intelligence and Games.

[7]  Mark J. Nelson.  Game Metrics Without Players: Strategies for Understanding Game Artifacts , 2011, Artificial Intelligence in the Game Design Process.

[8]  S. LaValle.  Rapidly-exploring random trees : a new tool for path planning , 1998 .

[9]  Kenneth O. Stanley,et al.  Go-Explore: a New Approach for Hard-Exploration Problems , 2019, ArXiv.

[10]  Liming Zhu,et al.  Software quality and agile methods , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[11]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[12]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[13]  Philip Bachman,et al.  Deep Reinforcement Learning that Matters , 2017, AAAI.

[14]  Michael Mateas,et al.  Computational Support for Play Testing Game Sketches , 2009, AIIDE.

[15]  Annie Louis,et al.  Beyond playing to win: Diversifying heuristics for GVGAI , 2017, 2017 IEEE Conference on Computational Intelligence and Games (CIG).

[16]  Frederic P. Miller,et al.  Super Nintendo Entertainment System , 2009 .

[17]  Peter L. Bartlett,et al.  RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning , 2016, ArXiv.

[18]  Adam M. Smith,et al.  Retrieving Game States with Moment Vectors , 2018, AAAI Workshops.

[19]  Kenneth I. Magel,et al.  Examining the Effectiveness of Testing Coverage Tools : An Empirical Study , 2014 .

[20]  Frédéric Maire,et al.  A Framework for the Semi-Automatic Testing of Video Games , 2008, AIIDE.

[21]  Rainer Malaka,et al.  Automated Game Testing with ICARUS: Intelligent Completion of Adventure Riddles via Unsupervised Solving , 2017, CHI PLAY.

[22]  Julian Togelius,et al.  Automated Playtesting With Procedural Personas Through MCTS With Evolved Heuristics , 2018, IEEE Transactions on Games.