A Wavelet-based Encoding for Neuroevolution

A new indirect scheme for encoding neural network connection weights as sets of wavelet-domain coefficients is proposed in this paper. It exploits spatial regularities in the weight-space to reduce the gene-space dimension by considering the low-frequency wavelet coefficients only. The wavelet-based encoding builds on top of a frequency-domain encoding, but unlike when using a Fourier-type transform, it offers gene locality while preserving continuity of the genotype-phenotype mapping. We argue that this added property allows for more efficient evolutionary search and demonstrate this on the octopus-arm control task, where superior solutions were found in fewer generations. The scalability of the wavelet-based encoding is shown by evolving networks with many parameters to control game-playing agents in the Arcade Learning Environment.

[1]  Risto Miikkulainen,et al.  A Neuroevolution Approach to General Atari Game Playing , 2014, IEEE Transactions on Computational Intelligence and AI in Games.

[2]  Aria Nosratinia,et al.  Wavelet-Based Image Coding: An Overview , 1999 .

[3]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[4]  Christian Jacob,et al.  Genetic L-System Programming , 1994, PPSN.

[5]  Martin A. Riedmiller.  Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method , 2005, ECML.

[6]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[7]  Jürgen Schmidhuber,et al.  Evolving neural networks in compressed weight space , 2010, GECCO '10.

[8]  Jürgen Schmidhuber,et al.  Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability , 1997, Neural Networks.

[9]  John Holland,et al.  Adaptation in Natural and Artificial Sys-tems: An Introductory Analysis with Applications to Biology , 1975 .

[10]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[11]  C. R. Reeves,et al.  Evolving Neural Feedforward Networks , 1993 .

[12]  Risto Miikkulainen,et al.  Effective image compression using evolved wavelets , 2005, GECCO '05.

[13]  Henry Tabe,et al.  Wavelet Transform , 2009, Encyclopedia of Biometrics.

[14]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[15]  Joël M. H. Karel,et al.  Optimal discrete wavelet design for cardiac signal processing , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[16]  A. Haar.  Zur Theorie der orthogonalen Funktionensysteme , 1910 .

[17]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[18]  Jürgen Schmidhuber,et al.  Evolving large-scale neural networks for vision-based reinforcement learning , 2013, GECCO '13.

[19]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents , 2012, J. Artif. Intell. Res..

[20]  Tamar Flash,et al.  Dynamic model of the octopus arm. I. Biomechanics of the octopus reaching movement. , 2005, Journal of neurophysiology.

[21]  Kenneth O. Stanley,et al.  Generating large-scale neural networks through discovering geometric regularities , 2007, GECCO '07.

[22]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[23]  Tom Schaul,et al.  Natural Evolution Strategies , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[24]  Jürgen Schmidhuber,et al.  Compressed Network Complexity Search , 2012, PPSN.

[25]  Tamar Flash,et al.  Dynamic model of the octopus arm. II. Control of reaching movements. , 2005, Journal of neurophysiology.

[26]  Y. Meyer,et al.  Wavelets and Filter Banks , 1991 .

[27]  Risto Miikkulainen,et al.  Accelerated Neural Evolution through Cooperatively Coevolved Synapses , 2008, J. Mach. Learn. Res..

[28]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[29]  Tom Schaul,et al.  High dimensions and heavy tails for natural evolution strategies , 2011, GECCO '11.

[30]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[31]  Ingrid Daubechies,et al.  The wavelet transform, time-frequency localization and signal analysis , 1990, IEEE Trans. Inf. Theory.

[32]  J. Urgen Schmidhuber.  Discovering Problem Solutions with Low Kolmogorov Complexity and High Generalization Capability , 1994 .

[33]  Charles E. Taylor.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. Complex Adaptive Systems.John H. Holland , 1994 .

[34]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[35]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.