ESAE: Evolutionary Strategy-Based Architecture Evolution

Although deep neural networks (DNNs) play important roles in many fields, the architecture design of DNNs can be challenging due to the difficulty of input data representation, the huge number of parameters and the complex layer relationships. To overcome the obstacles of architecture design, we developed a new method to generate the optimal structure of DNNs, named Evolutionary Strategy-based Architecture Evolution (ESAE), consisting of a bi-level representation and a probability distribution learning approach. The bi-level representation encodes architectures in the gene and parameter levels. The probability distribution learning approach ensures the efficient convergence of the architecture searching process. By using Fashion-MNIST and CIFAR-10, the effectiveness of the proposed ESAS is verified. The evolved DNNs, starting from a trivial initial architecture with one single convolutional layer, achieved the accuracies of 94.48% and 93.49% on Fashion-MNIST and CIFAR-10, respectively, and require remarkably less hardware costs in terms of GPUs and running time, compared with the existing state-of-the-art manual screwed architectures.

[1]  Shenquan Wang,et al.  Distributed H∞ consensus control for nonlinear multi-agent systems under switching topologies via relative output feedback , 2017, Neural Computing and Applications.

[2]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[3]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Alok Aggarwal,et al.  Regularized Evolution for Image Classifier Architecture Search , 2018, AAAI.

[5]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[6]  Yiming Yang,et al.  DARTS: Differentiable Architecture Search , 2018, ICLR.

[7]  Quoc V. Le,et al.  Large-Scale Evolution of Image Classifiers , 2017, ICML.

[8]  Michael R. Lyu,et al.  A hybrid particle swarm optimization-back-propagation algorithm for feedforward neural network training , 2007, Appl. Math. Comput..

[9]  Kenneth O. Stanley,et al.  A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks , 2009, Artificial Life.

[10]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[11]  Prabhat,et al.  Scalable Bayesian Optimization Using Deep Neural Networks , 2015, ICML.

[12]  Alok Aggarwal,et al.  Aging Evolution for Image Classifier Architecture Search , 2019, AAAI 2019.

[13]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  George Kurian,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[15]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[17]  S. Deb,et al.  Elephant Herding Optimization , 2015, 2015 3rd International Symposium on Computational and Business Intelligence (ISCBI).

[18]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[19]  Gregory D. Hager,et al.  sharpDARTS: Faster and More Accurate Differentiable Architecture Search , 2019, ArXiv.

[20]  Moncef Gabbouj,et al.  Evolutionary artificial neural networks by multi-dimensional particle swarm optimization , 2009, Neural Networks.

[21]  Masanori Suganuma,et al.  A genetic programming approach to designing convolutional neural network architectures , 2017, GECCO.

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  Gaige Wang,et al.  Moth search algorithm: a bio-inspired metaheuristic algorithm for global optimization problems , 2016, Memetic Computing.

[24]  Khalid M. Salama,et al.  Learning neural network structures with ant colony algorithms , 2015, Swarm Intelligence.

[25]  Nuno Lourenço,et al.  DENSER: deep evolutionary network structured representation , 2018, Genetic Programming and Evolvable Machines.

[26]  Prasant Kumar Pattnaik,et al.  Artificial Neural Network trained by Particle Swarm Optimization for non-linear channel equalization , 2014, Expert Syst. Appl..

[27]  Min-Xia Zhang,et al.  Shallow and deep neural network training by water wave optimization , 2019, Swarm Evol. Comput..

[28]  Elliot Meyerson,et al.  Evolving Deep Neural Networks , 2017, Artificial Intelligence in the Age of Neural Networks and Brain Computing.

[29]  Leandro dos Santos Coelho,et al.  Earthworm optimisation algorithm: a bio-inspired metaheuristic algorithm for global optimisation problems , 2018, Int. J. Bio Inspired Comput..

[30]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[31]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[32]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[33]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Zhihua Cui,et al.  Monarch butterfly optimization , 2015, Neural Computing and Applications.