Block Diagonal Natural Evolution Strategies

The Natural Evolution Strategies (NES) family of search algorithms have been shown to be efficient black-box optimizers, but the most powerful version xNES does not scale to problems with more than a few hundred dimensions. And the scalable variant, SNES, potentially ignores important correlations between parameters. This paper introduces Block Diagonal NES (BD-NES), a variant of NES which uses a block diagonal covariance matrix. The resulting update equations are computationally effective on problems with much higher dimensionality than their full-covariance counterparts, while retaining faster convergence speed than methods that ignore covariance information altogether. The algorithm has been tested on the Octopus-arm benchmark, and the experiments section presents performance statistics showing that BD-NES achieves better performance than SNES on networks that are too large to be optimized by xNES.

[1]  Tom Schaul,et al.  Exponential natural evolution strategies , 2010, GECCO '10.

[2]  Faustino J. Gomez,et al.  Intrinsically Motivated Evolutionary Search for Vision-Based Reinforcement Learning , 2011 .

[3]  John F. Kolen,et al.  Evaluating Benchmark Problems by Random Guessing , 2001 .

[4]  Y. Engel,et al.  A dynamical model of the octopus arm , 1997, Neuroscience Letters.

[5]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[6]  Albert-László Barabási,et al.  Statistical mechanics of complex networks , 2001, ArXiv.

[7]  Tom Schaul,et al.  Stochastic search using the natural gradient , 2009, ICML '09.

[8]  Jurgen Schmidhuber,et al.  Intrinsically motivated neuroevolution for vision-based reinforcement learning , 2011, 2011 IEEE International Conference on Development and Learning (ICDL).

[9]  Robert Schaefer.  Parallel Problem Solving from Nature - PPSN XI, 11th International Conference, Kraków, Poland, September 11-15, 2010. Proceedings, Part II , 2010, PPSN.

[10]  Mitchell A. Potter,et al.  EVOLVING NEURAL NETWORKS WITH COLLABORATIVE SPECIES , 2006 .

[11]  Tom Schaul,et al.  High dimensions and heavy tails for natural evolution strategies , 2011, GECCO '11.

[12]  Risto Miikkulainen,et al.  Incremental Evolution of Complex General Behavior , 1997, Adapt. Behav..

[13]  Julian F. Miller,et al.  Genetic and Evolutionary Computation — GECCO 2003 , 2003, Lecture Notes in Computer Science.

[14]  Kenneth O. Stanley,et al.  Evolving a Single Scalable Controller for an Octopus Arm with a Variable Number of Segments , 2010, PPSN.

[15]  Tamar Flash,et al.  Dynamic model of the octopus arm. I. Biomechanics of the octopus reaching movement. , 2005, Journal of neurophysiology.

[16]  Mark W. Spong,et al.  Swing up control of the Acrobot , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[17]  Tom Schaul,et al.  Natural Evolution Strategies , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).