NeuroEvolution: Evolving Heterogeneous Artificial Neural Networks

NeuroEvolution is the application of Evolutionary Algorithms to the training of Artificial Neural Networks. Currently the vast majority of NeuroEvolutionary methods create homogeneous networks of user defined transfer functions. This is despite NeuroEvolution being capable of creating heterogeneous networks where each neuron’s transfer function is not chosen by the user, but selected or optimised during evolution. This paper demonstrates how NeuroEvolution can be used to select or optimise each neuron’s transfer function and empirically shows that doing so significantly aids training. This result is important as the majority of NeuroEvolutionary methods are capable of creating heterogeneous networks using the methods described.

[1]  Yoshua Bengio,et al.  Exploring Strategies for Training Deep Neural Networks , 2009, J. Mach. Learn. Res..

[2]  Julian Francis Miller,et al.  Cartesian Genetic Programming , 2015, Cartesian Genetic Programming.

[3]  Tao Xiong,et al.  A combined SVM and LDA approach for classification , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[4]  Wlodzislaw Duch,et al.  Transfer functions: hidden possibilities for better neural networks , 2001, ESANN.

[5]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[6]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[7]  P. Husbands,et al.  Incremental Evolution of Neural Network Architectures for Adaptive Behaviour Incremental Evolution of Neural Network Architectures for Adaptive Behaviour , 1993 .

[8]  Hod Lipson,et al.  Comparison of tree and graph encodings as function of problem complexity , 2007, GECCO '07.

[9]  Wlodzislaw Duch,et al.  Make it cheap: Learning with O(nd) complexity , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[10]  Norbert Jankowski,et al.  Survey of Neural Transfer Functions , 1999 .

[11]  Ernesto Costa,et al.  Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories , 2009, Genetic Programming and Evolvable Machines.

[12]  Xin Yao,et al.  Evolutionary design of artificial neural networks with different nodes , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[13]  Julian Francis Miller,et al.  Cartesian Genetic Programming: Why No Bloat? , 2014, EuroGP.

[14]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[15]  Julian Francis Miller,et al.  Cartesian genetic programming , 2010, GECCO.

[16]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[17]  Marijke F. Augusteijn,et al.  Evolving transfer functions for artificial neural networks , 2003, Neural Computing & Applications.

[18]  R. Poli,et al.  Discovery of Symbolic, Neuro-Symbolic and Neural Networks with Parallel Distributed Genetic Programming , 1997, ICANNGA.

[19]  Abdennasser Chebira,et al.  A Neural Network Based Approach For Sensors Issued Data Fusion , 2003 .

[20]  E. Cantu-Paz,et al.  An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[21]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[22]  Julian Francis Miller,et al.  The Importance of Topology Evolution in NeuroEvolution: A Case Study Using Cartesian Genetic Programming of Artificial Neural Networks , 2013, SGAI Conf..

[23]  Julian Francis Miller,et al.  The Advantages of Landscape Neutrality in Digital Circuit Evolution , 2000, ICES.

[24]  Riccardo Poli,et al.  Parallel Distributed Genetic Programming , 1996 .

[25]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[26]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[27]  Thomas F. Coleman,et al.  Large-Scale Numerical Optimization , 1990 .

[28]  Eric Van Wyk,et al.  Evolution of internal dynamics for neural network nodes , 2009, Evol. Intell..

[29]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[30]  Julian F. Miller,et al.  What bloat? Cartesian Genetic Programming on Boolean problems , 2003 .

[31]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[32]  Jürgen Schmidhuber,et al.  Evolving neural networks in compressed weight space , 2010, GECCO '10.

[33]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[34]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[35]  F.J. Von Zuben,et al.  Hierarchical evolution of heterogeneous neural networks , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[36]  O. Mangasarian,et al.  Pattern Recognition Via Linear Programming: Theory and Application to Medical Diagnosis , 1989 .

[37]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[38]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[39]  Julian Francis Miller,et al.  Cartesian genetic programming encoded artificial neural networks: a comparison using three benchmarks , 2013, GECCO '13.

[40]  Julian Francis Miller,et al.  Recurrent Cartesian Genetic Programming , 2014, PPSN.

[41]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[42]  Rajkumar Roy,et al.  Advances in Soft Computing , 2018, Lecture Notes in Computer Science.

[43]  Gul Muhammad Khan,et al.  Fast learning neural networks using Cartesian genetic programming , 2013, Neurocomputing.

[44]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[45]  Katsushi Ikeuchi,et al.  Symbolic visual learning , 1997 .

[46]  Stephan K. Chalup,et al.  Variations of the two-spiral task , 2007, Connect. Sci..

[47]  Julian Francis Miller,et al.  Redundancy and computational efficiency in Cartesian genetic programming , 2006, IEEE Transactions on Evolutionary Computation.

[48]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[49]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[50]  A. P. Wieland,et al.  Evolving neural network controllers for unstable systems , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[51]  Julian Francis Miller,et al.  Neutrality and the Evolvability of Boolean Function Landscape , 2001, EuroGP.

[52]  Sebastian Thrun,et al.  The MONK''s Problems-A Performance Comparison of Different Learning Algorithms, CMU-CS-91-197, Sch , 1991 .

[53]  Paul Walsh,et al.  Improving the Performance of CGPANN for Breast Cancer Diagnosis Using Crossover and Radial Basis Functions , 2013, EvoBIO.