Online Build-Order Optimization for Real-Time Strategy Agents using Multi-Objective Evolutionary Algorithms

Abstract : The investigation introduces a novel approach for online build-order optimization in real-time strategy (RTS) games. The goal of our research is to develop an arti cial intelligence (AI) RTS planning agent for military critical decision- making education with the ability to perform at an expert human level, as well as to assess a players critical decision- making ability or skill-level. Build-order optimization is modeled as a multi-objective problem (MOP), and solutions are generated utilizing a multi-objective evolutionary algorithm (MOEA) that provides a set of good build-orders to a RTS planning agent. We de ne three research objectives: (1) Design, implement and validate a capability to determine the skill-level of a RTS player. (2) Design, implement and validate a strategic planning tool that produces near expert level build-orders which are an ordered sequence of actions a player can issue to achieve a goal, and (3) Integrate the strategic planning tool into our existing RTS agent framework and an RTS game engine. The skill-level metric we selected provides an original and needed method of evaluating a RTS players skill-level during game play. This metric is a high-level description of how quickly a player executes a strategy versus known players executing the same strategy. Our strategic planning tool combines a game simulator and an MOEA to produce a set of diverse and good build-orders for an RTS agent. Through the integration of case-base reasoning (CBR), planning goals are derived and expert build- orders are injected into a MOEA population. The MOEA then produces a diverse and approximate Pareto front that is integrated into our AI RTS agent framework. Thus, the planning tool provides an innovative online approach for strategic planning in RTS games. Experimentation via the Spring Engine Balanced Annihilation game reveals that the strategic planner is able to discover build-orders that are better than an expert scripted agent and thus ac

[1]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[2]  Michael Mateas,et al.  Case-Based Reasoning for Build Order in Real-Time Strategy Games , 2009, AIIDE.

[3]  James F. Allen.  Maintaining knowledge about temporal intervals , 1983, CACM.

[4]  P. Baptiste,et al.  Edge-Finding Constraint Propagation Algorithms for Disjunctive and Cumulative Scheduling , 1996 .

[5]  Santiago Ontañón,et al.  Stochastic Plan Optimization in Real-Time Strategy Games , 2008, AIIDE.

[6]  Michael Buro,et al.  Adversarial Planning Through Strategy Simulation , 2007, 2007 IEEE Symposium on Computational Intelligence and Games.

[7]  M. Buro,et al.  A FIRST LOOK AT BUILD-ORDER OPTIMIZATION IN REAL-TIME STRATEGY GAMES , 2007 .

[8]  Kalyanmoy Deb,et al.  Multi-objective Genetic Algorithms: Problem Difficulties and Construction of Test Problems , 1999, Evolutionary Computation.

[9]  John E. Laird,et al.  SORTS: A Human-Level Approach to Real-Time Strategy AI , 2007, AIIDE.

[10]  David W. Aha,et al.  Learning to Win: Case-Based Plan Selection in a Real-Time Strategy Game , 2005, Künstliche Intell..

[11]  Santiago Ontañón,et al.  Case-Based Planning and Execution for Real-Time Strategy Games , 2007, ICCBR.

[12]  Ashwin Ram,et al.  Transfer Learning in Real-Time Strategy Games Using Hybrid CBR/RL , 2007, IJCAI.

[13]  Arnav Jhala,et al.  Learning from Demonstration for Goal-Driven Autonomy , 2012, AAAI.

[14]  Pieter Spronck,et al.  Opponent Modeling in Real-Time Strategy Games , 2007, GAMEON.

[15]  J. Platt.  Sequential Minimal Optimization : A Fast Algorithm for Training Support Vector Machines , 1998 .

[16]  David P. Williamson,et al.  The Design of Approximation Algorithms , 2011 .

[17]  William J Cojocar.  Adaptive Leadership in the Military Decision Making Process , 2012 .

[18]  Enrique Alba,et al.  A Study of Convergence Speed in Multi-objective Metaheuristics , 2008, PPSN.

[19]  Jeffrey N. Rule.  A Symbiotic Relationship: The Observe-Orient-Decide-Act (OODA) Loop, Intuition, and Strategic Thought , 2013 .

[21]  Günter Rudolph,et al.  Multi-objective assessment of pre-optimized build orders exemplified for StarCraft 2 , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[22]  Helmut Simonis,et al.  Modelling Producer/Consumer Constraints , 1995, CP.

[23]  Jill Fain Lehman,et al.  A Gentle Introduction to Soar, an Architecture for Human Cognition. , 1996 .

[24]  El-Ghazali Talbi,et al.  Metaheuristics - From Design to Implementation , 2009 .

[25]  Di Trapani,et al.  A Real-time Strategy Agent Framework and Strategy Classifier for Computer Generated Forces , 2012 .

[26]  Alan Fern,et al.  Online Planning for Resource Production in Real-Time Strategy Games , 2007, ICAPS.

[27]  Pierre Bessière,et al.  A Bayesian Model for Plan Recognition in RTS Games Applied to StarCraft , 2011, AIIDE.

[28]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[29]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.

[30]  Michael Mateas,et al.  Conceptual Neighborhoods for Retrieval in Case-Based Reasoning , 2009, ICCBR.

[31]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[32]  Qingfu Zhang,et al.  Analyzing limits of effectiveness in different implementations of estimation of distribution algorithms , 2011 .

[33]  Tetske Avontuur.  Modeling player skill in Starcraft II , 2012 .

[34]  Luiz Chaimowicz,et al.  An Artificial Intelligence System to Help the Player of Real-Time Strategy Games , 2010, 2010 Brazilian Symposium on Games and Digital Entertainment.

[35]  Pat Langley,et al.  A Unified Cognitive Architecture for Physical Agents , 2006, AAAI.

[36]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[37]  Ben George Weber,et al.  Integrating Learning in a Multi-Scale Agent , 2012 .

[38]  Michael Buro,et al.  Incorporating Search Algorithms into RTS Game Agents , 2012 .

[39]  Arnav Jhala,et al.  Using Data Mining to Model Player Experience , 2011 .

[40]  Michael Buro,et al.  Real-Time Strategy Games: A New AI Research Challenge , 2003, IJCAI.

[41]  Arnav Jhala,et al.  Reactive planning idioms for multi-scale game AI , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[42]  Sushil J. Louis,et al.  Finding robust strategies to defeat specific opponents using case-injected coevolution , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[43]  Alan Fern,et al.  Extending Online Planning for Resource Production in Real-Time Strategy Games with Search , 2007 .

[44]  Tom Bylander,et al.  Complexity Results for Planning , 1991, IJCAI.

[45]  Philippe Baptiste.  Disjunctive constraints for manufacturing scheduling: Principles and extensions , 1996 .

[46]  Alan Fern,et al.  UCT for Tactical Assault Planning in Real-Time Strategy Games , 2009, IJCAI.

[47]  Teresita Sotomayor,et al.  Evaluating tactical combat casualty care training treatments effects on combat medic trainees in light of select human descriptive characteristics , 2008 .

[48]  C. Mariano,et al.  MOAQ an Ant-Q algorithm for multiple objective optimization problems , 1999 .

[49]  Michael Mateas,et al.  A data mining approach to strategy prediction , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[50]  Jonathan Schaeffer,et al.  Monte Carlo Planning in RTS Games , 2005, CIG.

[51]  Michael Mateas,et al.  An Integrated Agent for Playing Real-Time Strategy Games , 2008, AAAI.

[52]  Michèle Sebag,et al.  Multi-objective Monte-Carlo Tree Search , 2012, ACML.

[53]  David E. Goldberg,et al.  Multi-objective bayesian optimization algorithm , 2002 .

[54]  Harald Köstler,et al.  Optimizing Opening Strategies in a Real-time Strategy Game by a Multi-objective Genetic Algorithm , 2012, SGAI Conf..

[55]  Michael Buro,et al.  Build Order Optimization in StarCraft , 2011, AIIDE.