Multi-Context Generation in Virtual Reality Environments Using Deep Reinforcement Learning

In this work, a Deep Reinforcement Learning (RL) approach is proposed for Procedural Content Generation (PCG) that seeks to automate the generation of multiple related virtual reality (VR) environments for enhanced personalized learning. This allows for the user to be exposed to multiple virtual scenarios that demonstrate a consistent theme, which is especially valuable in an educational context. RL approaches to PCG offer the advantage of not requiring training data, as opposed to other PCG approaches that employ supervised learning approaches. This work advances the state of the art in RL-based PCG by demonstrating the ability to generate a diversity of contexts in order to teach the same underlying concept. A case study is presented that demonstrates the feasibility of the proposed RL-based PCG method using examples of probability distributions in both manufacturing facility and grocery store virtual environments. The method demonstrated in this paper has the potential to enable the automatic generation of a variety of virtual environments that are connected by a common concept or theme.

[1]  E. Mandinach,et al.  The role of cognitive engagement in classroom learning and motivation , 1983 .

[2]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[3]  Jennifer A. Bennett,et al.  A Virtual Tour of the Cell: Impact of Virtual Reality on Student Learning and Engagement in the STEM Classroom † , 2019, Journal of microbiology & biology education.

[4]  Pebertli Nils Alho Barata,et al.  Consolidating Learning in Power Systems: Virtual Reality Applied to the Study of the Operation of Electric Power Transformers , 2015, IEEE Transactions on Education.

[5]  Zoran Popovic,et al.  A case study of expressively constrainable level design automation tools for a puzzle game , 2012, FDG.

[6]  Brendan Iribe Oculus Rift를 이용한 체감형 게임 구현 , 2014 .

[7]  Julian Togelius,et al.  Towards Player Adaptivity in a Serious Game for Conflict Resolution , 2011, 2011 Third International Conference on Games and Virtual Worlds for Serious Applications.

[8]  Georgios N. Yannakakis Game AI revisited , 2012, CF '12.

[9]  Rafael Bidarra,et al.  Integrating semantics and procedural generation: key enabling factors for declarative modeling of virtual worlds , 2010 .

[10]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[11]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[12]  J. Star,et al.  Using Digital Resources for Motivation and Engagement in Learning Mathematics: Reflections from Teachers and Students , 2016 .

[13]  Rafael Rieder,et al.  Procedural Content Generation using Artificial Intelligence for Unique Virtual Reality Game Experiences , 2019, 2019 21st Symposium on Virtual and Augmented Reality (SVR).

[14]  Moslem Yousefi,et al.  A systematic review of data-driven approaches in player modeling of educational games , 2017, Artificial Intelligence Review.

[15]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[16]  Julian Togelius,et al.  Deep Reinforcement Learning for General Video Game AI , 2018, 2018 IEEE Conference on Computational Intelligence and Games (CIG).

[17]  Beryl Plimmer,et al.  A systematic review of Virtual Reality in education , 2017 .

[18]  Seyed M. Buhari,et al.  Towards simulation of the classroom learning experience: Virtual reality approach , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[19]  Michele D. Dickey Brave new (interactive) worlds: A review of the design affordances and constraints of two 3D virtual worlds as interactive learning environments , 2005, Interact. Learn. Environ..

[20]  Jonathan P. Rowe,et al.  Interactive Narrative Personalization with Deep Reinforcement Learning , 2017, IJCAI.

[21]  Nathan R Sturtevant,et al.  Deep Static and Dynamic Level Analysis: A Study on Infinite Mario , 2016 .

[22]  Chris Dede,et al.  Situated Learning in Virtual Worlds and Immersive Simulations , 2014 .

[23]  Morteza Behrooz,et al.  The Learning of Zelda: Data-Driven Learning of Level Topology , 2015 .

[24]  Mark O. Riedl,et al.  Mixed Reality Meets Procedural Content Generation in Video Games , 2021, Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment.

[25]  Ke Chen,et al.  Learning Constructive Primitives for Real-Time Dynamic Difficulty Adjustment in Super Mario Bros , 2018, IEEE Transactions on Games.

[26]  Minhong Wang,et al.  A data-driven procedural-content-generation approach for educational games , 2018, J. Comput. Assist. Learn..

[27]  Tassos A. Mikropoulos,et al.  Educational virtual environments: A ten-year review of empirical research (1999-2009) , 2011, Comput. Educ..

[28]  Onur Çalişkan,et al.  Virtual field trips in education of earth and environmental sciences , 2011 .

[29]  Murat Akcay,et al.  Advantages and challenges associated with augmented reality for education : A systematic review of the literature , 2017 .

[30]  Flemming Konradsen,et al.  A review of the use of virtual reality head-mounted displays in education and training , 2017, Education and Information Technologies.

[31]  Simon M. Lucas,et al.  Evolving mario levels in the latent space of a deep convolutional generative adversarial network , 2018, GECCO.

[32]  Julian Togelius,et al.  Deep Learning for Video Game Playing , 2017, IEEE Transactions on Games.

[33]  Conrad S. Tucker,et al.  Reinforcement Learning Content Generation for Virtual Reality Applications , 2019, Volume 1: 39th Computers and Information in Engineering Conference.

[34]  Luiz A. L. Rodrigues,et al.  A Math Educacional Computer Game Using Procedural Content Generation , 2017 .

[35]  Michael Mateas,et al.  Super Mario as a String: Platformer Level Generation Via LSTMs , 2016, DiGRA/FDG.

[36]  Julian Togelius,et al.  The 2010 Mario AI Championship: Level Generation Track , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[37]  Michael Mateas,et al.  Scenario generation for emergency rescue training games , 2009, FDG.

[38]  Michele H. Miller,et al.  Meeting the NAE Grand Challenge: Personalized Learning for Engineering Students through Instruction on Metacognition and Motivation Strategies , 2013 .

[39]  Marwan Mattar,et al.  Unity: A General Platform for Intelligent Agents , 2018, ArXiv.

[40]  Michela Ott,et al.  A LITERATURE REVIEW ON IMMERSIVE VIRTUAL REALITY IN EDUCATION: STATE OF THE ART AND PERSPECTIVES. , 2015, 11th International Conference eLearning and Software for Education.

[41]  Julian Togelius,et al.  Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation , 2018, 1806.10729.

[42]  William Winn,et al.  When Does Immersion in a Virtual Environment Help Students Construct Understanding , 2002 .

[43]  Xin Xu,et al.  Reinforcement learning algorithms with function approximation: Recent advances and applications , 2014, Inf. Sci..

[44]  Daniel A. Guttentag Virtual reality: Applications and implications for tourism , 2010 .

[45]  Michael Mateas,et al.  Tanagra: Reactive Planning and Constraint Solving for Mixed-Initiative Level Design , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[46]  Julian Togelius,et al.  Fully Differentiable Procedural Content Generation through Generative Playing Networks , 2020, ArXiv.

[47]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[48]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[49]  Seok Hee Oh,et al.  A Study on Creation and Usability of Real Time City Generator via Procedural Content Generation: – Focus on virtual reality contents for senior , 2019, 2019 International Symposium on Multimedia and Communication Technology (ISMAC).

[50]  Wadee S. Alhalabi,et al.  Virtual reality systems enhance students’ achievements in engineering education , 2016, Behav. Inf. Technol..

[51]  Moslem Yousefi,et al.  A Procedural Content Generation-Based Framework for Educational Games: Toward a Tailored Data-Driven Game for Developing Early English Reading Skills , 2018 .

[52]  Sang Do Noh,et al.  Virtual reality applications in manufacturing industries: Past research, present findings, and future directions , 2015, Concurr. Eng. Res. Appl..

[53]  Hugo Paredes,et al.  Exploring educational immersive videogames: an empirical study with a 3D multimodal interaction prototype , 2016, Behav. Inf. Technol..

[54]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[55]  Julian Togelius,et al.  A comparative evaluation of procedural level generators in the Mario AI framework , 2014, FDG.

[56]  Ingrid Isenhardt,et al.  Towards Measuring User Experience, Activation and Task Performance in Immersive Virtual Learning Environments for Students , 2016, iLRN.

[57]  Kristin L. K. Koskey,et al.  Motivation, learning, and transformative experience: A study of deep engagement in science , 2010 .