Rating the skill of synthetic agents in competitive multi-agent environments

A very effective and promising approach to simulate real-life conditions in multi-agent virtual environments with intelligent agents is to introduce social parameters and dynamics. Introduction of social parameters in such settings reshapes the overall performance of the synthetic agents, so a new challenge of reconsidering the methods to assess agents’ evolution emerges. In a number of studies regarding such environments, the rating of the agents is being considered in terms of metrics (or measures or simple grading) designed for humans, such as Elo and Glicko. In this paper, we present a large-scale evaluation of existing rating methods and a proposal for a new rating approach named Relative Skill-Level Estimator (RSLE), which can be regarded as a base for developing rating systems for multi-agent systems. The presented comparative study highlights an inconsistency in rating synthetic agents in the context described by the widely used methods and demonstrates the efficiency of the new RSLE.

[1]  Belal Al-Khateeb,et al.  Introducing a Round Robin Tournament into Evolutionary Individual and Social Learning Checkers , 2011, 2011 Developments in E-systems Engineering.

[2]  Chih-Jen Lin,et al.  A Bayesian Approximation Method for Online Ranking , 2011, J. Mach. Learn. Res..

[3]  Tshilidzi Marwala,et al.  Social Learning methods in board game agents , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[4]  A. Elo.  The rating of chessplayers, past and present , 1978 .

[5]  H. D. Rombach,et al.  The Goal Question Metric Approach , 1994 .

[6]  Chairi Kiourt,et al.  How game complexity affects the playing behavior of synthetic agents , 2017, EUMAS/AT.

[7]  Hal S. Stern,et al.  Designing a College Football Playoff System , 1999 .

[8]  Bart De Schutter,et al.  A Comprehensive Survey of Multiagent Reinforcement Learning , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[9]  Daniel Kudenko,et al.  Reinforcement learning of coordination in cooperative multi-agent systems , 2002, AAAI/IAAI.

[10]  Dimitrios Kalles,et al.  Measuring Expert Impact on Learning how to Play a Board Game , 2007, AIAI.

[11]  Tom Minka,et al.  TrueSkillTM: A Bayesian Skill Rating System , 2006, NIPS.

[12]  Sean Luke,et al.  Cooperative Multi-Agent Learning: The State of the Art , 2005, Autonomous Agents and Multi-Agent Systems.

[13]  Chairi Kiourt,et al.  Human Rating Methods on Multi-agent Systems , 2015, EUMAS/AT.

[14]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[15]  Gerald Tesauro,et al.  Temporal difference learning and TD-Gammon , 1995, CACM.

[16]  Sergey I. Nikolenko,et al.  A New Bayesian Rating System for Team Competitions , 2011, ICML.

[17]  Chairi Kiourt,et al.  Social Reinforcement Learning in Game Playing , 2012, 2012 IEEE 24th International Conference on Tools with Artificial Intelligence.

[18]  G. Nigel Gilbert,et al.  Simulation for the social scientist , 1999 .

[19]  J. March.  Exploration and exploitation in organizational learning , 1991, STUDI ORGANIZZATIVI.

[20]  Chairi Kiourt,et al.  Synthetic learning agents in game-playing social environments , 2016, Adapt. Behav..

[21]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[22]  David J. Hand,et al.  Who's #1? The science of rating and ranking , 2012 .

[23]  Yoav Shoham,et al.  Multiagent Systems - Algorithmic, Game-Theoretic, and Logical Foundations , 2009 .

[24]  C. Spearman.  The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[25]  Antonio F. Gómez-Skarmeta,et al.  Using cognitive agents in social simulations , 2011, Eng. Appl. Artif. Intell..

[26]  Rüdiger Dillmann,et al.  Integrating skills into multi-agent systems , 1998, J. Intell. Manuf..

[27]  Jacques Ferber,et al.  Multi-agent systems - an introduction to distributed artificial intelligence , 1999 .

[28]  Dorian Kodelja,et al.  Multiagent cooperation and competition with deep reinforcement learning , 2015, PloS one.

[29]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[30]  Kagan Tumer,et al.  Elo Ratings for Structural Credit Assignment in Multiagent Systems , 2013, AAAI.

[31]  G. Tesauro.  Practical Issues in Temporal Difference Learning , 1992 .

[32]  Pierpaolo Di Bitonto,et al.  An Evaluation Method for Multi-Agent Systems , 2010, KES-AMSTA.

[33]  Steven S. Beauchemin,et al.  Workflow Nets for Multiagent Cooperation , 2012, IEEE Transactions on Automation Science and Engineering.

[34]  Lea Kutvonen,et al.  Reputation Management Survey , 2007, The Second International Conference on Availability, Reliability and Security (ARES'07).

[35]  Chairi Kiourt,et al.  ReSkill: Relative Skill-Level Calculation System , 2016, SETN.

[36]  Lucy A. Suchman,et al.  Making work visible , 1995, CACM.

[37]  Jacques Ferber,et al.  From Agents to Organizations: An Organizational View of Multi-agent Systems , 2003, AOSE.

[38]  Mooweon Rhee,et al.  Exploration and Exploitation , 2016 .

[39]  John Tromp,et al.  SOLVING CONNECT-4 ON MEDIUM BOARD SIZES , 2008 .

[40]  Stefan Edelkamp,et al.  Symbolic Classification of General Two-Player Games , 2008, KI.

[41]  Weisong Shi,et al.  Performance evaluation of rating aggregation algorithms in reputation systems , 2005, 2005 International Conference on Collaborative Computing: Networking, Applications and Worksharing.

[42]  Gerald Tesauro,et al.  Programming backgammon using self-teaching neural nets , 2002, Artif. Intell..

[43]  Ricardo Calvo.  Valencia Spain: The Cradle of European Chess , 2007 .

[44]  António J. M. Castro,et al.  Multi-Agent Learning in both Cooperative and Competitive Environments , 2013 .

[45]  M. Glickman.  Parameter Estimation in Large Dynamic Paired Comparison Experiments , 1999 .

[46]  Karl Tuyls,et al.  An Overview of Cooperative and Competitive Multiagent Learning , 2005, LAMAS.

[47]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[48]  Simon P. Attle,et al.  Cooperative Learning in a Competitive Environment: Classroom Applications , 2007 .

[49]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[50]  Dimitrios Kalles,et al.  On verifying game designs and playing strategies using reinforcement learning , 2001, SAC.