An Adaptive Markov Game Model for Threat Intent Inference

In an adversarial military environment, it is important to efficiently and promptly predict the enemy's tactical intent from lower level spatial and temporal information. In this paper, we propose a decentralized Markov game (MG) theoretic approach to estimate the belief of each possible enemy course of action (ECOA), which is utilized to model the adversary intents. It has the following advantages: (1) It is decentralized. Each cluster or team makes decisions mostly based on local information. We put more autonomies in each group allowing for more flexibilities; (2) A Markov decision process (MDP) can effectively model the uncertainties in the noisy military environment; (3) It is a game model with three players: red force (enemies), blue force (friendly forces), and white force (neutral objects); (4) Correlated-Q reinforcement learning is integrated. With the consideration that actual value functions are not normally known and they must be estimated, we integrate correlated-Q learning concept in our game approach to dynamically adjust the payoffs function of each player. A simulation software package has been developed to demonstrate the performance of our proposed algorithms. Simulations have verified that our proposed algorithms are scalable, stable, and satisfactory in performance.

[1]  S. Sastry.  Nonlinear Systems: Analysis, Stability, and Control , 1999 .

[2]  T. Başar,et al.  Dynamic Noncooperative Game Theory , 1982 .

[3]  John J. Salerno,et al.  A situation awareness model applied to multiple domains , 2005, SPIE Defense + Commercial Sensing.

[4]  Gang Tao,et al.  Adaptive Control Design and Analysis , 2003 .

[5]  J. B. Cruz,et al.  Moving horizon Nash strategies for a military air operation , 2002 .

[6]  Michael L. Littman,et al.  Friend-or-Foe Q-learning in General-Sum Games , 2001, ICML.

[7]  Keith B. Hall,et al.  Correlated Q-Learning , 2003, ICML.

[8]  Dan Shen,et al.  Nash strategies for dynamic noncooperative linear quadratic sequential games , 2007, 2007 46th IEEE Conference on Decision and Control.

[9]  J. Cruz,et al.  On the Stackelberg strategy in nonzero-sum games , 1973 .

[10]  John J. Salerno,et al.  Realizing situation awareness within a cyber environment , 2006, SPIE Defense + Commercial Sensing.

[11]  Svein J. Knapskog,et al.  Using Stochastic Game Theory to Compute the Expected Behavior of Attackers , 2005 .

[12]  Genshe Chen,et al.  A Game Theoretic Approach to Mission Planning For Multiple Aerial Platforms , 2005 .

[13]  T. Basar,et al.  A game theoretic approach to decision and analysis in network intrusion detection , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[14]  Joao P. Hespanha,et al.  Deception in Non-Cooperative Games with Partial Information , 2000 .

[15]  Nicolas Vieille,et al.  Correlated Equilibrium in Stochastic Games , 2002, Games Econ. Behav..

[16]  Dongxu Li,et al.  Team dynamics and tactics for mission planning , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[17]  D. Fudenberg,et al.  The Theory of Learning in Games , 1998 .

[18]  Jose B. Cruz,et al.  Game Theoretic Approach to Threat Prediction and Situation Awareness , 2006, 2006 9th International Conference on Information Fusion.

[19]  Karl Johan Åström,et al.  Adaptive Control , 1989, Embedded Digital Control with Microcontrollers.

[20]  Ming Li,et al.  Game-theoretic modeling and control of a military air operation , 2001 .

[21]  Michael P. Wellman,et al.  Multiagent Reinforcement Learning: Theoretical Framework and an Algorithm , 1998, ICML.

[22]  L. Shapley,et al.  Stochastic Games* , 1953, Proceedings of the National Academy of Sciences.

[23]  E. Rowland.  Theory of Games and Economic Behavior , 1946, Nature.

[24]  T. Başar,et al.  Dynamic Noncooperative Game Theory, 2nd Edition , 1998 .

[25]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[26]  K. Basu,et al.  A non-cooperative game approach for intrusion detection in sensor networks , 2004, IEEE 60th Vehicular Technology Conference, 2004. VTC2004-Fall. 2004.