Differential Evolution for Neural Networks Optimization

In this paper, a Neural Networks optimizer based on Self-adaptive Differential Evolution is presented. This optimizer applies mutation and crossover operators in a new way, taking into account the structure of the network according to a per layer strategy. Moreover, a new crossover called interm is proposed, and a new self-adaptive version of DE called MAB-ShaDE is suggested to reduce the number of parameters. The framework has been tested on some well-known classification problems and a comparative study on the various combinations of self-adaptive methods, mutation, and crossover operators available in literature is performed. Experimental results show that DENN reaches good performances in terms of accuracy, better than or at least comparable with those obtained by backpropagation.

[1]  Verena Heidrich-Meisner,et al.  Neuroevolution strategies for episodic reinforcement learning , 2009, J. Algorithms.

[2]  Janez Brest,et al.  Self-Adapting Control Parameters in Differential Evolution: A Comparative Study on Numerical Benchmark Problems , 2006, IEEE Transactions on Evolutionary Computation.

[3]  Yajun Wang,et al.  Combinatorial Multi-Armed Bandit and Its Extension to Probabilistically Triggered Arms , 2014, J. Mach. Learn. Res..

[4]  Joni-Kristian Kämäräinen,et al.  Differential Evolution Training Algorithm for Feed-Forward Neural Networks , 2003, Neural Processing Letters.

[5]  Frederico G. Guimarães,et al.  Self-adaptive mutation in the differential evolution , 2011, GECCO '11.

[6]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[7]  N. Schraudolph,et al.  Dynamic Parameter Encoding for Genetic Algorithms , 1992, Machine Learning.

[8]  D. Quade.  Using Weighted Rankings in the Analysis of Complete Blocks with Additive Block Effects , 1979 .

[9]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[10]  Jianhua Wu,et al.  A modified differential evolution algorithm for unconstrained optimization problems , 2013, Neurocomputing.

[11]  Marcin Andrychowicz,et al.  Neural Random Access Machines , 2015, ERCIM News.

[12]  Fei Peng,et al.  Multi-start JADE with knowledge transfer for numerical optimization , 2009, IEEE Congress on Evolutionary Computation.

[13]  Alex S. Fukunaga,et al.  Improving the search performance of SHADE using linear population size reduction , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[14]  Valentina Franzoni,et al.  A Multistrain Bacterial Diffusion Model for Link Prediction , 2017, Int. J. Pattern Recognit. Artif. Intell..

[15]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[16]  Xiaodong Li,et al.  Time series forecasting by evolving artificial neural networks with genetic algorithms, differential evolution and estimation of distribution algorithm , 2011, Neural Computing and Applications.

[17]  Tao Chen,et al.  Back propagation neural network with adaptive differential evolution algorithm for time series forecasting , 2015, Expert Syst. Appl..

[18]  René Thomsen,et al.  A comparative study of differential evolution, particle swarm optimization, and evolutionary algorithms on numerical benchmark problems , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[19]  Songmin Jia,et al.  Differential-Evolution-Based Coevolution Ant Colony Optimization Algorithm for Bayesian Network Structure Learning , 2018, Algorithms.

[20]  Masao Arakawa,et al.  Differential evolution as the global optimization technique and its application to structural optimization , 2011, Appl. Soft Comput..

[21]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[22]  Adam P. Piotrowski,et al.  Differential Evolution algorithms applied to Neural Network training suffer from stagnation , 2014, Appl. Soft Comput..

[23]  Alfredo Milani,et al.  A Differential Evolution Algorithm for the Permutation Flowshop Scheduling Problem with Total Flow Time Criterion , 2014, PPSN.

[24]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[25]  Marco Baioletti,et al.  Can Differential Evolution Be an Efficient Engine to Optimize Neural Networks? , 2017, MOD.

[26]  Kenneth O. Stanley,et al.  Simple Evolutionary Optimization Can Rival Stochastic Gradient Descent in Neural Networks , 2016, GECCO.

[27]  Amit Konar,et al.  Differential Evolution Using a Neighborhood-Based Mutation Operator , 2009, IEEE Transactions on Evolutionary Computation.

[28]  Valentina Franzoni,et al.  A Deep Learning Semantic Approach to Emotion Recognition Using the IBM Watson Bluemix Alchemy Language , 2017, ICCSA.

[29]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[30]  Ausif Mahmood,et al.  Differential Evolution: A Survey and Analysis , 2018, Applied Sciences.

[31]  R. Storn,et al.  Differential Evolution: A Practical Approach to Global Optimization (Natural Computing Series) , 2005 .

[32]  Luca Maria Gambardella,et al.  Deep, Big, Simple Neural Nets for Handwritten Digit Recognition , 2010, Neural Computation.

[33]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[34]  Alex S. Fukunaga,et al.  Success-history based parameter adaptation for Differential Evolution , 2013, 2013 IEEE Congress on Evolutionary Computation.

[35]  Alfredo Milani,et al.  Linear Ordering Optimization with a Combinatorial Differential Evolution , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[36]  Alex Graves,et al.  Neural Turing Machines , 2014, ArXiv.

[37]  Dario Floreano,et al.  Center of mass encoding: a self-adaptive representation with adjustable redundancy for real-valued parameters , 2007, GECCO '07.

[38]  Kenneth O. Stanley,et al.  Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning , 2017, ArXiv.

[39]  H. Khanna Nehemiah,et al.  Neural network classifier optimization using Differential Evolution with Global Information and Back Propagation algorithm for clinical datasets , 2016, Appl. Soft Comput..

[40]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[41]  Ponnuthurai N. Suganthan,et al.  Recent advances in differential evolution - An updated survey , 2016, Swarm Evol. Comput..

[42]  Navdeep Jaitly,et al.  Hybrid speech recognition with Deep Bidirectional LSTM , 2013, 2013 IEEE Workshop on Automatic Speech Recognition and Understanding.

[43]  Alfredo Milani,et al.  Detecting Hate Speech for Italian Language in Social Media , 2018, EVALITA@CLiC-it.

[44]  W. Land,et al.  A new training algorithm for the general regression neural network , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[45]  Alfredo Milani,et al.  Self regulating mechanisms for network immunization , 2016, AI Commun..

[46]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.