On the automated, evolutionary design of neural networks: past, present, and future

Neuroevolution is the name given to a field of computer science that applies evolutionary computation for evolving some aspects of neural networks. After the AI Winter came to an end, neural networks reemerged to solve a great variety of problems. However, their usage requires designing their topology, a decision with a potentially high impact on performance. Whereas many works have tried to suggest rules-of-thumb for designing topologies, the truth is that there are not analytic procedures for determining the optimal one for a given problem, and trial-and-error is often used instead. Neuroevolution arose almost 3 decades ago, with some works focusing on the evolutionary design of the topology and most works describing techniques for learning connection weights. Since then, evolutionary computation has been proved to be a convenient approach for determining the topology and weights of neural networks, and neuroevolution has been applied to a great variety of fields. However, for more than 2 decades neuroevolution has mainly focused on simple artificial neural networks models, far from today’s deep learning standards. This is insufficient for determining good architectures for modern networks extensively used nowadays, which involve multiple hidden layers, recurrent cells, etc. More importantly, deep and convolutional neural networks have become a de facto standard in representation learning for solving many different problems, and neuroevolution has only focused in this kind of networks in very recent years, with many works being presented in 2017 onward. In this paper, we review the field of neuroevolution during the last 3 decades. We will put the focus on very recent works on the evolution of deep and convolutional neural networks, which is a new but growing field of study. To the best of our knowledge, this is the best survey reviewing the literature in this field, and we have described the features of each work as well as their performance on well-known databases when available. This work aims to provide a complete reference of all works related to neuroevolution of convolutional neural networks up to the date. Finally, we will provide some future directions for the advancement of this research area.

[1]  Hui Li,et al.  Evolutionary artificial neural networks: a review , 2011, Artificial Intelligence Review.

[2]  Michael Scholz.  A Learning Strategy for Neural Networks Based on a Modified Evolutionary Strategy , 1990, PPSN.

[3]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[4]  John Tran,et al.  cuDNN: Efficient Primitives for Deep Learning , 2014, ArXiv.

[5]  Nikolaus Hansen,et al.  The CMA Evolution Strategy: A Comparing Review , 2006, Towards a New Evolutionary Computation.

[6]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[7]  Christian Igel,et al.  Neuroevolution for reinforcement learning using evolution strategies , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[8]  Masanori Suganuma,et al.  A genetic programming approach to designing convolutional neural network architectures , 2017, GECCO.

[9]  Yoshua Bengio,et al.  An empirical evaluation of deep architectures on problems with many factors of variation , 2007, ICML '07.

[10]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[11]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Amit Choudhary,et al.  Influence of Introducing an Additional Hidden Layer on the Character Recognition Capability of a BP Neural Network having One Hidden Layer , 2010 .

[13]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[14]  Razvan Pascanu,et al.  Theano: A CPU and GPU Math Compiler in Python , 2010, SciPy.

[15]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[16]  Yamashita,et al.  Backpropagation algorithm which varies the number of hidden units , 1989 .

[17]  Kenneth O. Stanley,et al.  A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks , 2009, Artificial Life.

[18]  D. R. McGregor,et al.  Designing application-specific neural networks using the structured genetic algorithm , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[19]  Peter J. B. Hancock,et al.  Genetic algorithms and permutation problems: a comparison of recombination operators for neural net structure specification , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[20]  Hitoshi Iba,et al.  Evolutionary Approach to Deep Learning , 2018 .

[21]  Kenneth O. Stanley,et al.  Quality Diversity: A New Frontier for Evolutionary Computation , 2016, Front. Robot. AI.

[22]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[23]  Nichael Lynn Cramer,et al.  A Representation for the Adaptive Generation of Simple Sequential Programs , 1985, ICGA.

[24]  Jiri Matas,et al.  Systematic evaluation of convolution neural network advances on the Imagenet , 2017, Comput. Vis. Image Underst..

[25]  John R. Koza,et al.  Hierarchical Genetic Algorithms Operating on Populations of Computer Programs , 1989, IJCAI.

[26]  Christian Blum,et al.  Metaheuristics in combinatorial optimization: Overview and conceptual comparison , 2003, CSUR.

[27]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[28]  Kenneth O. Stanley,et al.  Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning , 2017, ArXiv.

[29]  Jan Torreele,et al.  Temporal Processing with Recurrent Networks: An Evolutionary Approach , 1991, ICGA.

[30]  Neil D. Lawrence,et al.  International Conference on Learning Representations Workshop track , 2016 .

[31]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[32]  L. Darrell Whitley,et al.  Optimizing Neural Networks Using FasterMore Accurate Genetic Search , 1989, ICGA.

[33]  Oliver Kramer,et al.  Evolution of Convolutional Highway Networks , 2017, EvoApplications.

[34]  L. Baird.  Reinforcement Learning Through Gradient Descent , 1999 .

[35]  L. Darrell Whitley,et al.  Genetic Reinforcement Learning with Multilayer Neural Networks , 1991, ICGA.

[36]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[37]  J. G. Elias.  Genetic generation of connection patterns for a dynamic artificial neural network , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[38]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[39]  S. Linnainmaa.  Taylor expansion of the accumulated rounding error , 1976 .

[40]  Alan L. Yuille,et al.  Genetic CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[41]  Lakhmi C. Jain,et al.  Automatic Generation of Neural Network Architecture Using Evolutionary Computation , 1997, Advances in Fuzzy Systems - Applications and Theory.

[42]  Alejandro Baldominos Gómez,et al.  Evolutionary Design of Convolutional Neural Networks for Human Activity Recognition in Sensor-Rich Environments , 2018, Sensors.

[43]  Oliver Kramer,et al.  Lamarckian Evolution of Convolutional Neural Networks , 2018, PPSN.

[44]  David D. Cox,et al.  Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , 2013, ICML.

[45]  Danielle S. Bassett,et al.  Learning, Memory, and the Role of Neural Network Architecture , 2011, PLoS Comput. Biol..

[46]  Michael C. Mozer,et al.  Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment , 1988, NIPS.

[47]  Eugenio Culurciello,et al.  An Analysis of Deep Neural Network Models for Practical Applications , 2016, ArXiv.

[48]  Alok Aggarwal,et al.  Regularized Evolution for Image Classifier Architecture Search , 2018, AAAI.

[49]  Travis Desell,et al.  Large scale evolution of convolutional neural networks using volunteer computing , 2017, GECCO.

[50]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[51]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[52]  Julian Morris,et al.  A procedure for determining the topology of multilayer feedforward neural networks , 1994, Neural Networks.

[53]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[54]  Geoffrey I. Webb,et al.  Encyclopedia of Machine Learning and Data Mining , 2017, Encyclopedia of Machine Learning and Data Mining.

[55]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[56]  Gerald Sommer,et al.  Evolutionary reinforcement learning of artificial neural networks , 2007, Int. J. Hybrid Intell. Syst..

[57]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[58]  Josh Harguess,et al.  Image Classification Using Generative Neuro Evolution for Deep Learning , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[59]  J. M. Smith,et al.  Optimization Theory in Evolution , 1978 .

[60]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[61]  Marvin Minsky,et al.  An introduction to computational geometry , 1969 .

[62]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[63]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[64]  Larry J. Eshelman,et al.  Using genetic search to exploit the emergent behavior of neural networks , 1990 .

[65]  Steven R. Young,et al.  Evolving Deep Networks Using HPC , 2017, MLHPC@SC.

[66]  David J. Chalmers,et al.  The Evolution of Learning: An Experiment in Genetic Connectionism , 1991 .

[67]  Shahid Ali,et al.  Evolving deep neural networks: A new prospect , 2016, 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD).

[68]  El-Ghazali Talbi,et al.  Metaheuristics - From Design to Implementation , 2009 .

[69]  Josh Harguess,et al.  Generative NeuroEvolution for Deep Learning , 2013, ArXiv.

[70]  Vasant Honavar,et al.  Evolutionary Design of Neural Architectures -- A Preliminary Taxonomy and Guide to Literature , 1995 .

[71]  David Pfau,et al.  Convolution by Evolution: Differentiable Pattern Producing Networks , 2016, GECCO.

[72]  Quoc V. Le,et al.  Large-Scale Evolution of Image Classifiers , 2017, ICML.

[73]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[74]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[75]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[76]  Risto Miikkulainen,et al.  Accelerated Neural Evolution through Cooperatively Coevolved Synapses , 2008, J. Mach. Learn. Res..

[77]  Lakhmi C. Jain,et al.  Automatic generation of a neural network architecture using evolutionary computation , 1995, Proceedings Electronic Technology Directions to the Year 2000.

[78]  Risto Miikkulainen,et al.  Topology of a Neural Network , 2017, Encyclopedia of Machine Learning and Data Mining.

[79]  Mengjie Zhang,et al.  Evolving Deep Convolutional Neural Networks for Image Classification , 2017, IEEE Transactions on Evolutionary Computation.

[80]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[81]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[82]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[83]  Tariq Samad,et al.  Designing Application-Specific Neural Networks Using the Genetic Algorithm , 1989, NIPS.

[84]  J. M. Smith,et al.  Optimality theory in evolutionary biology , 1990, Nature.

[85]  Darrell Whitley,et al.  Genetic cascade learning for neural networks , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[86]  John R. Koza,et al.  Genetic Programming: The Movie , 1992 .

[87]  Arend Hintze,et al.  Integrated Information Increases with Fitness in the Evolution of Animats , 2011, PLoS Comput. Biol..

[88]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[89]  Oriol Vinyals,et al.  Hierarchical Representations for Efficient Architecture Search , 2017, ICLR.

[90]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[91]  Jürgen Schmidhuber,et al.  Evolving deep unsupervised convolutional networks for vision-based reinforcement learning , 2014, GECCO.

[92]  Robert F. Port,et al.  Fractally configured neural networks , 1991, Neural Networks.

[93]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[94]  Alejandro Baldominos Gómez,et al.  Model Selection in Committees of Evolved Convolutional Neural Networks Using Genetic Algorithms , 2018, IDEAL.

[95]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[96]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[97]  C. Lee Giles,et al.  What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation , 1998 .

[98]  Elliot Meyerson,et al.  Evolving Deep Neural Networks , 2017, Artificial Intelligence in the Age of Neural Networks and Brain Computing.

[99]  Gerald Sommer,et al.  Efficient reinforcement learning through Evolutionary Acquisition of Neural Topologies , 2005, ESANN.

[100]  Hitoshi Iba.  Evolutionary Approach to Machine Learning and Deep Neural Networks , 2018, Springer Singapore.

[101]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[102]  Richard S. Forsyth,et al.  BEAGLE—A DARWINIAN APPROACH TO PATTERN RECOGNITION , 1981 .

[103]  Wolfram Schiffmann,et al.  Performance Evaluation of Evolutionarily Created Neural Network Topologies , 1990, PPSN.

[104]  Alejandro Baldominos Gómez,et al.  Evolutionary convolutional neural networks: An application to handwriting recognition , 2017, Neurocomputing.

[105]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[106]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[107]  Frank Kirchner,et al.  A common genetic encoding for both direct and indirect encodings of networks , 2007, GECCO '07.

[108]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[109]  K. Gnana Sheela,et al.  Review on Methods to Fix Number of Hidden Neurons in Neural Networks , 2013 .

[110]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[111]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[112]  Frank Hutter,et al.  CMA-ES for Hyperparameter Optimization of Deep Neural Networks , 2016, ArXiv.

[113]  Kalyanmoy Deb,et al.  NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search , 2018, ArXiv.

[114]  K. Lindgren,et al.  Regular language inference using evolving neural networks , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[115]  Tobias Senst,et al.  Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[116]  Yoshua Bengio,et al.  On the Properties of Neural Machine Translation: Encoder–Decoder Approaches , 2014, SSST@EMNLP.

[117]  Steven R. Young,et al.  Optimizing deep learning hyper-parameters through an evolutionary algorithm , 2015, MLHPC@SC.

[118]  Oliver Kramer,et al.  Limited Evaluation Evolutionary Optimization of Large Neural Networks , 2018, KI.

[119]  Nuno Lourenço,et al.  DENSER: deep evolutionary network structured representation , 2018, Genetic Programming and Evolvable Machines.

[120]  Zachary Chase Lipton.  A Critical Review of Recurrent Neural Networks for Sequence Learning , 2015, ArXiv.

[121]  Wei Zhang,et al.  Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks , 2018, NeurIPS.

[122]  Mengjie Zhang,et al.  Automatically Evolving CNN Architectures Based on Blocks , 2018, ArXiv.

[123]  Dušan Petrovački,et al.  Evolutional development of a multilevel neural network , 1993, Neural Networks.

[124]  Tariq Samad,et al.  Towards the Genetic Synthesisof Neural Networks , 1989, ICGA.

[125]  Sebastian Risi,et al.  An Enhanced Hypercube-Based Encoding for Evolving the Placement, Density, and Connectivity of Neurons , 2012, Artificial Life.

[126]  Bin Wang,et al.  A Hybrid DE Approach to Designing CNN for Image Classification , 2018 .