Evolutionary Dynamics of Ant Colony Optimization

Swarm intelligence has been successfully applied in various domains, e.g., path planning, resource allocation and data mining. Despite its wide use, a theoretical framework in which the behavior of swarm intelligence can be formally understood is still lacking. This article starts by formally deriving the evolutionary dynamics of ant colony optimization, an important swarm intelligence algorithm. We then continue to formally link these to reinforcement learning. Specifically, we show that the attained evolutionary dynamics are equivalent to the dynamics of Q-learning. Both algorithms are equivalent to a dynamical system known as the replicator dynamics in the domain of evolutionary game theory. In conclusion, the process of improvement described by the replicator dynamics appears to be a fundamental principle which drives processes in swarm intelligence, evolution, and learning.

[1]  Thomas Stützle,et al.  Ant colony optimization: artificial ants as a computational intelligence technique , 2006 .

[2]  D. E. Matthews.  Evolution and the Theory of Games , 1977 .

[3]  R. Michod,et al.  John Maynard Smith. , 2005, Annual review of genetics.

[4]  Defang Liu,et al.  Biological Swarm Intelligence Based Opportunistic Resource Allocation for Wireless Ad Hoc Networks , 2012, Wirel. Pers. Commun..

[5]  Peter Vrancx,et al.  Networks of Learning Automata and Limiting Games , 2007, Adaptive Agents and Multi-Agents Systems.

[6]  Peter Stone,et al.  RoboCup 2000: Robot Soccer World Cup IV , 2001, RoboCup.

[7]  Ajith Abraham,et al.  Swarm Intelligence in Data Mining , 2009, Swarm Intelligence in Data Mining.

[8]  Aram Galstyan,et al.  Dynamics of Softmax Q-Learning in Two-Player Two-Action Games , 2011, ArXiv.

[9]  Karl Tuyls,et al.  FAQ-Learning in Matrix Games: Demonstrating Convergence Near Nash Equilibria, and Bifurcation of Attractors in the Battle of Sexes , 2011, Interactive Decision Theory and Game Theory.

[10]  Walter J. Gutjahr,et al.  A Graph-based Ant System and its convergence , 2000, Future Gener. Comput. Syst..

[11]  Karl Tuyls,et al.  Frequency adjusted multi-agent Q-learning , 2010, AAMAS.

[12]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[13]  Jing Zhou,et al.  Swarm Intelligence: Ant-Based Robot Path Planning , 2009, 2009 Fifth International Conference on Information Assurance and Security.

[14]  Josef Hofbauer,et al.  Evolutionary Games and Population Dynamics , 1998 .

[15]  W. Gutjahr.  On the Finite-Time Dynamics of Ant Colony Optimization , 2006 .

[16]  Tilman Börgers,et al.  Learning Through Reinforcement and Replicator Dynamics , 1997 .

[17]  Simon Parsons,et al.  What evolutionary game theory tells us about multiagent learning , 2007, Artif. Intell..

[18]  Aram Galstyan,et al.  Dynamics of Boltzmann Q learning in two-player two-action games. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[19]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[20]  Thomas Stützle,et al.  Ant Colony Optimization , 2009, EMO.

[21]  Karl Tuyls,et al.  An Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games , 2005, Autonomous Agents and Multi-Agent Systems.

[22]  Stefan Schaal,et al.  2008 Special Issue: Reinforcement learning of motor skills with policy gradients , 2008 .

[23]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[24]  Peter Vrancx,et al.  Switching dynamics of multi-agent learning , 2008, AAMAS.

[25]  M. Dorigo,et al.  1 Positive Feedback as a Search Strategy , 1991 .

[26]  Martin Middendorf,et al.  Modeling the Dynamics of Ant Colony Optimization , 2002, Evolutionary Computation.

[27]  Sridhar Mahadevan,et al.  Automatic Programming of Behavior-Based Robots Using Reinforcement Learning , 1991, Artif. Intell..

[28]  Christopher M. Vigorito.  Distributed path planning for mobile robots using a swarm of interacting reinforcement learners , 2007, AAMAS '07.

[29]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[30]  Matthias Rauterberg,et al.  State-coupled replicator dynamics , 2009, AAMAS.

[31]  Daniel Kudenko,et al.  Adaptive Agents and Multi-Agent Systems II: Adaptation and Multi-Agent Learning , 2003, Adaptive Agents and Multi-Agent Systems.

[32]  Marco Dorigo,et al.  Ant system: optimization by a colony of cooperating agents , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[33]  Günther Palm,et al.  Evolutionary stable strategies and game dynamics for n-person games , 1984 .