Evolving programs to build artificial neural networks

In general, the topology of Artificial Neural Networks (ANNs) is human-engineered and learning is merely the process of weight adjustment. However, it is well known that this can lead to sub-optimal solutions. Topology and Weight Evolving Artificial Neural Networks (TWEANNs) can lead to better topologies however, once obtained they remain fixed and cannot adapt to new problems. In this chapter, rather than evolving a fixed structure artificial neural network as in neuroevolution, we evolve a pair of programs that build the network. One program runs inside neurons and allows them to move, change, die or replicate. The other is executed inside dendrites and allows them to change length and weight, be removed, or replicate. The programs are represented and evolved using Cartesian Genetic Programming. From the developed networks multiple traditional ANNs can be extracted, each of which solves a different problem. The proposed approach has been evaluated on multiple classification problems.

[1]  J. Kleim,et al.  Synaptogenesis and FOS Expression in the Motor Cortex of the Adult Rat after Motor Skill Learning , 1996, The Journal of Neuroscience.

[2]  William F. Punch,et al.  Reducing Wasted Evaluations in Cartesian Genetic Programming , 2013, EuroGP.

[3]  Kenneth O. Stanley,et al.  Compositional Pattern Producing Networks : A Novel Abstraction of Development , 2007 .

[4]  Risto Miikkulainen,et al.  Efficient evolution of neural network topologies , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[5]  P. Rakic,et al.  Principles of neural cell migration , 1990, Experientia.

[6]  Jean-Baptiste Mouret,et al.  Evolving neural networks that are both modular and regular: HyperNEAT plus the connection cost technique , 2014, GECCO.

[7]  Tinne Tuytelaars,et al.  Expert Gate: Lifelong Learning with a Network of Experts , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Richard S. J. Frackowiak,et al.  Navigation-related structural change in the hippocampi of taxi drivers. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[9]  R. French,et al.  Catastrophic Forgetting in Connectionist Networks: Causes, Consequences and Solutions , 1994 .

[10]  Gul Muhammad Khan,et al.  Evolution of Cartesian Genetic Programs for Development of Learning Neural Architecture , 2011, Evolutionary Computation.

[11]  Chris Adami,et al.  Evolution of Robust Developmental Neural Networks , 2004 .

[12]  Julian F. Miller,et al.  Evolving Developmental Programs That Build Neural Networks for Solving Multiple Problems , 2018, GPTP.

[13]  F. Valverde,et al.  Rate and extent of recovery from dark rearing in the visual cortex of the mouse. , 1971, Brain research.

[14]  A. Dekaban,et al.  Changes in brain weights during the span of human life: Relation of brain weights to body heights and body weights , 1978, Annals of neurology.

[15]  Eliot A. Brenowitz,et al.  Seasonal plasticity in the adult brain , 2000, Trends in Neurosciences.

[16]  Sebastian Risi,et al.  Enhancing es-hyperneat to evolve more complex regular neural networks , 2011, GECCO '11.

[17]  Pasko Rakic,et al.  Modes and Mishaps of Neuronal Migration in the Mammalian Brain , 2008, The Journal of Neuroscience.

[18]  Frédéric Gruau,et al.  Automatic Definition of Modular Neural Networks , 1994, Adapt. Behav..

[19]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[20]  Michael O'Neill,et al.  Grammatical Evolution: Evolving Programs for an Arbitrary Language , 1998, EuroGP.

[21]  Keith L. Downing,et al.  Supplementing evolutionary developmental systems with abstract models of neurogenesis , 2007, GECCO '07.

[22]  Jeff Clune,et al.  A novel generative encoding for evolving modular, regular and scalable networks , 2011, GECCO '11.

[23]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[24]  Christoph Adami,et al.  A Developmental Model for the Evolution of Artificial Neural Networks , 2000, Artificial Life.

[25]  A. Isles.  Neural and behavioral epigenetics; what it is, and what is hype , 2015, Genes, brain, and behavior.

[26]  Alistair G. Rust,et al.  Evolutionary neural topiary: growing and sculpting artifical neurons to order , 2000 .

[27]  Eric J. Nestler,et al.  Epigenetic regulation in psychiatric disorders , 2007, Nature Reviews Neuroscience.

[28]  D. Ballard,et al.  Learning-Dependent Synaptic Modifications in the Cerebellar Cortex of the Adult Rat Persist for at Least Four Weeks , 1997, The Journal of Neuroscience.

[29]  A. Cangelosi,et al.  Cell division and migration in a 'genotype' for neural networks (Cell division and migration in neural networks) , 1993 .

[30]  Peter J. Bentley,et al.  On growth, form and computers , 2003 .

[31]  Julian Francis Miller,et al.  Cartesian genetic programming , 2000, GECCO '10.

[32]  Gul Muhammad Khan.  Evolution of Artificial Neural Development - In Search of Learning Genes , 2018, Studies in Computational Intelligence.

[33]  William F. Punch,et al.  Analysis of Cartesian Genetic Programming’s Evolutionary Mechanisms , 2015, IEEE Transactions on Evolutionary Computation.

[34]  Julian F. Miller,et al.  What bloat? Cartesian Genetic Programming on Boolean problems , 2003 .

[35]  Senén Barro,et al.  Do we need hundreds of classifiers to solve real world classification problems? , 2014, J. Mach. Learn. Res..

[36]  Julian Francis Miller,et al.  Recurrent Cartesian Genetic Programming , 2014, PPSN.

[37]  Jan Drchal,et al.  Tree-Based Indirect Encodings for Evolutionary Development of Neural Networks , 2008, ICANN.

[38]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[39]  Sebastian Risi,et al.  Evolving the placement and density of neurons in the hyperneat substrate , 2010, GECCO '10.

[40]  Gul Muhammad Khan,et al.  In search of intelligence: evolving a developmental neuron capable of learning , 2014, Connect. Sci..

[41]  Alexander V. Terekhov,et al.  Knowledge Transfer in Deep Block-Modular Neural Networks , 2015, Living Machines.

[42]  Jean-Arcady Meyer,et al.  Evolution and development of neural controllers for locomotion, gradient-following, and obstacle-avoidance in artificial insects , 1998, IEEE Trans. Neural Networks.

[43]  Cândida Ferreira,et al.  Gene Expression Programming: Mathematical Modeling by an Artificial Intelligence , 2014, Studies in Computational Intelligence.

[44]  Sung Young Jung.  A Topographical Method for the Development of Neural Networks for Artificial Brain Evolution , 2005, Artificial Life.

[45]  Jordan B. Pollack,et al.  Creating High-Level Components with a Generative Representation for Body-Brain Evolution , 2002, Artificial Life.

[46]  Dario Floreano,et al.  Neural morphogenesis, synaptic plasticity, and evolution , 2001, Theory in Biosciences.

[47]  Simon M. Lucas,et al.  A comparison of matrix rewriting versus direct encoding for evolving neural networks , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[48]  Julian Francis Miller,et al.  A Developmental Method for Growing Graphs and Circuits , 2003, ICES.

[49]  Dario Floreano,et al.  Evolutionary morphogenesis for multi-cellular systems , 2007, Genetic Programming and Evolvable Machines.

[50]  Julian Francis Miller,et al.  Cartesian genetic programming encoded artificial neural networks: a comparison using three benchmarks , 2013, GECCO '13.

[51]  Dianne Langford,et al.  HIV and antiretroviral therapy in the brain: neuronal injury and repair , 2008, Nature Reviews Neuroscience.

[52]  Julian Francis Miller,et al.  Neutrality and the Evolvability of Boolean Function Landscape , 2001, EuroGP.

[53]  M. Teicher,et al.  Evolving Synaptic Plasticity with an Evolutionary Cellular Development Model , 2008, PloS one.

[54]  Julian Francis Miller,et al.  The Advantages of Landscape Neutrality in Digital Circuit Evolution , 2000, ICES.

[55]  Diego Federici,et al.  A regenerating spiking neural network , 2005, Neural Networks.

[56]  R Ratcliff,et al.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. , 1990, Psychological review.

[57]  Larry D. Pyeatt,et al.  A comparison between cellular encoding and direct encoding for genetic neural networks , 1996 .

[58]  Jürgen Schmidhuber,et al.  Evolving neural networks in compressed weight space , 2010, GECCO '10.

[59]  Kenneth O. Stanley,et al.  A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks , 2009, Artificial Life.

[60]  John Raymond Smythies.  The Dynamic Neuron , 2002 .

[61]  Peter Eggenberger.  Creation of Neural Networks Based on Developmental and Evolutionary Principles , 1997 .

[62]  Julian Francis Miller,et al.  Redundancy and computational efficiency in Cartesian genetic programming , 2006, IEEE Transactions on Evolutionary Computation.

[63]  B. Kastrup.  The Universe in Consciousness , 2018 .

[64]  C. Nelson,et al.  Brain Development and the Role of Experience in the Early Years. , 2009, Zero to three.

[65]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[66]  Arjen Van Ooyen,et al.  Modeling neural development , 2003 .

[67]  Risto Miikkulainen,et al.  A Taxonomy for Artificial Embryogeny , 2003, Artificial Life.

[68]  Sebastian Risi,et al.  Indirectly Encoding Neural Plasticity as a Pattern of Local Rules , 2010, SAB.

[69]  Amanda J. C. Sharkey,et al.  Combining Artificial Neural Nets: Ensemble and Modular Multi-Net Systems , 1999 .

[70]  Andy Balaam,et al.  Developmental Neural Networks for Agents , 2003, ECAL.

[71]  Richard K. Belew,et al.  Interposing an Ontogenetic Model Between Genetic Algorithms and Neural Networks , 1992, NIPS.

[72]  Leonardo Franco,et al.  Constructive Neural Networks , 2009, Constructive Neural Networks.

[73]  R. French.  Catastrophic forgetting in connectionist networks , 1999, Trends in Cognitive Sciences.

[74]  Julian Francis Miller,et al.  Developments in Cartesian Genetic Programming: self-modifying CGP , 2010, Genetic Programming and Evolvable Machines.

[75]  Jordan B. Pollack,et al.  TITLE : Generative Representations for the Automated Design of Modular Physical Robots , 2003 .

[76]  T. Deacon.  The Symbolic Species: The Co-evolution of Language and the Brain , 1998 .