A Levy flight-based grey wolf optimizer combined with back-propagation algorithm for neural network training

In the present study, a new algorithm is developed for neural network training by combining a gradient-based and a meta-heuristic algorithm. The new algorithm benefits from simultaneous local and global search, eliminating the problem of getting stuck in local optimum. For this purpose, first the global search ability of the grey wolf optimizer (GWO) is improved with the Levy flight, a random walk in which the jump size follows the Levy distribution, which results in a more efficient global search in the search space thanks to the long jumps. Then, this improved algorithm is combined with back propagation (BP) to use the advantages of enhanced global search ability of GWO and local search ability of BP algorithm in training neural network. The performance of the proposed algorithm has been evaluated by comparing it against a number of well-known meta-heuristic algorithms using twelve classification and function-approximation datasets.

[1]  Seyed Mohammad Ali Razavi,et al.  Application of Image Analysis and Artificial Neural Network to Predict Mass Transfer Kinetics and Color Changes of Osmotically Dehydrated Kiwifruit , 2011 .

[2]  Enrique Alba,et al.  Training Neural Networks with GA Hybrid Algorithms , 2004, GECCO.

[3]  Doreen Meier,et al.  Fundamentals Of Neural Networks Architectures Algorithms And Applications , 2016 .

[4]  John H. Holland,et al.  Computer programs that " evolve " in ways that resemble natural selection can solve complex problems even their creators do not fully understand , 2022 .

[5]  Sung-Bae Cho,et al.  An improved swarm optimized functional link artificial neural network (ISO-FLANN) for classification , 2012, J. Syst. Softw..

[6]  Seyed Mohammad Mirjalili.  How effective is the Grey Wolf optimizer in training multi-layer perceptrons , 2014, Applied Intelligence.

[7]  Hong Wang,et al.  Application of Artificial Neural Network Supported by BP and Particle Swarm Optimization Algorithm for Evaluating the Criticality Class of Spare Parts , 2007, Third International Conference on Natural Computation (ICNC 2007).

[8]  Hossein Nezamabadi-pour,et al.  BGSA: binary gravitational search algorithm , 2010, Natural Computing.

[9]  Alireza Rezazadeh,et al.  Artificial neural network training using a new efficient optimization algorithm , 2013, Appl. Soft Comput..

[10]  Russell C. Eberhart,et al.  A new optimizer using particle swarm theory , 1995, MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science.

[11]  Petros Koumoutsakos,et al.  Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[12]  Jianzhou Wang,et al.  A Novel Hybrid Evolutionary Algorithm Based on PSO and AFSA for Feedforward Neural Network Training , 2008, 2008 4th International Conference on Wireless Communications, Networking and Mobile Computing.

[13]  H. Szu.  Fast simulated annealing , 1987 .

[14]  Hossein Nezamabadi-pour,et al.  GSA: A Gravitational Search Algorithm , 2009, Inf. Sci..

[15]  Fardin Akhlaghian Tab,et al.  Classification of Rice Varieties Using Optimal Color and Texture Features and BP Neural Networks , 2011, 2011 7th Iranian Conference on Machine Vision and Image Processing.

[16]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[17]  Andries Petrus Engelbrecht,et al.  Cooperative learning in neural networks using particle swarm optimizers , 2000, South Afr. Comput. J..

[18]  Masoud Yaghini,et al.  A hybrid algorithm for artificial neural network training , 2013, Eng. Appl. Artif. Intell..

[19]  Christian Blum,et al.  Training feed-forward neural networks with ant colony optimization: an application to pattern classification , 2005, Fifth International Conference on Hybrid Intelligent Systems (HIS'05).

[20]  Emile H. L. Aarts,et al.  Simulated Annealing: Theory and Applications , 1987, Mathematics and Its Applications.

[21]  Andrew Lewis,et al.  Grey Wolf Optimizer , 2014, Adv. Eng. Softw..

[22]  Zong Woo Geem,et al.  A New Heuristic Optimization Algorithm: Harmony Search , 2001, Simul..

[23]  Rajandrea Sethi,et al.  Artificial neural network simulation of hourly groundwater levels in a coastal aquifer system of the Venice lagoon , 2012, Eng. Appl. Artif. Intell..

[24]  Mohd Herwan Sulaiman,et al.  An Application of Grey Wolf Optimizer for Solving Combined Economic Emission Dispatch Problems , 2014 .

[25]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[26]  Dervis Karaboga,et al.  Hybrid Artificial Bee Colony algorithm for neural network training , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[27]  Prasant Kumar Pattnaik,et al.  Artificial Neural Network trained by Particle Swarm Optimization for non-linear channel equalization , 2014, Expert Syst. Appl..

[28]  Dervis Karaboga,et al.  A novel clustering approach: Artificial Bee Colony (ABC) algorithm , 2011, Appl. Soft Comput..

[29]  K. Lee,et al.  A new meta-heuristic algorithm for continuous engineering optimization: harmony search theory and practice , 2005 .

[30]  Andrew Lewis,et al.  Let a biogeography-based optimizer train your Multi-Layer Perceptron , 2014, Inf. Sci..

[31]  Xin-She Yang,et al.  Multiobjective cuckoo search for design optimization , 2013, Comput. Oper. Res..

[32]  Michael R. Lyu,et al.  A hybrid particle swarm optimization-back-propagation algorithm for feedforward neural network training , 2007, Appl. Math. Comput..

[33]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[34]  Lale Özbakir,et al.  Training neural networks with harmony search algorithms for classification problems , 2012, Eng. Appl. Artif. Intell..

[35]  Masoud Yaghini,et al.  HIOPGA : A New Hybrid Metaheuristic Algorithm to Train Feedforward Neural Networks for Prediction , 2011 .

[36]  Wei Cai,et al.  Grey Wolf Optimizer for parameter estimation in surface waves , 2015 .

[37]  Teresa Bernarda Ludermir,et al.  Hybrid Training of Feed-Forward Neural Networks with Particle Swarm Optimization , 2006, ICONIP.

[38]  Seyed Mohammad Mirjalili,et al.  Designing evolutionary feedforward neural networks using social spider optimization algorithm , 2015, Neural Computing and Applications.

[39]  Madhumita Panda,et al.  A Hybrid Differential Evolution and Back-Propagation Algorithm for Feedforward Neural Network Training , 2013 .

[40]  Mohd Herwan Sulaiman,et al.  Using the gray wolf optimizer for solving optimal reactive power dispatch problem , 2015, Appl. Soft Comput..

[41]  Bilal Alatas,et al.  ACROA: Artificial Chemical Reaction Optimization Algorithm for global optimization , 2011, Expert Syst. Appl..

[42]  G. M. Komaki,et al.  Grey Wolf Optimizer algorithm for the two-stage assembly flow shop scheduling problem with release time , 2015, J. Comput. Sci..

[43]  Christian Blum,et al.  An ant colony optimization algorithm for continuous optimization: application to feed-forward neural network training , 2007, Neural Computing and Applications.

[44]  Paulo Cortez,et al.  Particle swarms for feedforward neural network training , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[45]  Ahmad Ayatollahi,et al.  An efficient neural network based method for medical image segmentation , 2014, Comput. Biol. Medicine.

[46]  J. Klafter,et al.  Introduction to the Theory of Lévy Flights , 2008 .

[47]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[48]  Darío Baptista,et al.  A survey of artificial neural network training tools , 2013, Neural Computing and Applications.

[49]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[50]  Blum,et al.  [IEEE Fifth International Conference on Hybrid Intelligent Systems (HIS\'05) - Rio de Janeiro, Brazil (2005.11.6-2005.11.9)] Fifth International Conference on Hybrid Intelligent Systems (HIS\'05) - Training feed-forward neural networks with ant colony optimization: an application to pattern classifi , 2005 .

[51]  Siti Zaiton Mohd Hashim,et al.  Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm , 2012, Appl. Math. Comput..

[52]  Mehdi Ramezani,et al.  Training feedforward neural networks using hybrid particle swarm optimization and gravitational search , 2013 .