Effects of Input Addition in Learning for Adaptive Games: Towards Learning with Structural Changes

Adaptive Games (AG) involve a controller agent that continuously feeds from player actions and game state to tweak a set of game parameters in order to maintain or achieve an objective function such as the flow measure defined by Csikszentmihalyi. This can be considered a Reinforcement Learning (RL) situation, so that classical Machine Learning (ML) approaches can be used. On the other hand, many games naturally exhibit an incremental gameplay where new actions and elements are introduced or removed progressively to enhance player’s learning curve or to introduce variety within the game. This makes the RL situation unusual because the controller agent input/output signature can change over the course of learning. In this paper, we get interested in this unusual “protean” learning situation (PL). In particular, we assess how the learner can rely on its past shapes and experience to keep improving among signature changes without needing to restart the learning from scratch on each change. We first develop a rigorous formalization of the PL problem. Then, we address the first elementary signature change: “input addition”, with Recurrent Neural Networks (RNNs) in an idealized PL situation. As a first result, we find that it is possible to benefit from prior learning in RNNs even if the past controller agent signature has less inputs. The use of PL in AG thus remains encouraged. Investigating output addition, input/output removal and translating these results to generic PL will be part of future works.

[1]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[2]  Heng Wang,et al.  Concept drift detection for streaming data , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[3]  Fabien Michel,et al.  Formalisation of metamorph Reinforcement Learning , 2018 .

[4]  Peta Wyeth,et al.  GameFlow: a model for evaluating player enjoyment in games , 2005, CIE.

[5]  Masayuki Yamamura,et al.  Multitask reinforcement learning on the distribution of MDPs , 2003, Proceedings 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automation. Computational Intelligence in Robotics and Automation for the New Millennium (Cat. No.03EX694).

[6]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[7]  Mark B. Ring.  Continual learning in reinforcement environments , 1995, GMD-Bericht.

[8]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[9]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[10]  Pierre Bessière,et al.  Teaching Bayesian behaviours to video game characters , 2003, Robotics Auton. Syst..

[11]  Peter Stone,et al.  Transfer Learning for Reinforcement Learning Domains: A Survey , 2009, J. Mach. Learn. Res..

[12]  Zhanxing Zhu,et al.  Reinforced Continual Learning , 2018, NeurIPS.

[13]  Darryl Charles,et al.  Modular Reinforcement Learning architectures for artificially intelligent agents in complex game environments , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[14]  Alessandro Lazaric,et al.  Transfer in Reinforcement Learning: A Framework and a Survey , 2012, Reinforcement Learning.

[15]  Alexey Tsymbal,et al.  The problem of concept drift: definitions and related work , 2004 .

[16]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[17]  Pieter Abbeel,et al.  Meta Learning Shared Hierarchies , 2017, ICLR.

[18]  Antonio Mora García,et al.  The Believability Gene in Virtual Bots , 2016, FLAIRS.

[19]  Yee Whye Teh,et al.  Distral: Robust multitask reinforcement learning , 2017, NIPS.

[20]  Jenova Chen,et al.  Flow in games (and everything else) , 2007, CACM.

[21]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[22]  Sergey Levine,et al.  Learning modular neural network policies for multi-task and multi-robot transfer , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[23]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[24]  Cédric Buche,et al.  Automatable Evaluation Method Oriented toward Behaviour Believability for Video Games , 2008, GAMEON.