A benchmark of recent population-based metaheuristic algorithms for multi-layer neural network training

Multi-layer neural networks (MLNNs) are extensively used in many industrial applications. Training is the crucial task for MLNNs. While gradient descent-based approaches are most commonly employed here, they suffer from drawbacks such as getting stuck in local optima. One approach to address this is to employ population-based metaheuristic algorithms. Since a variety of such algorithms have been proposed in the literature, in this paper we benchmark the performance of 15 metaheuristic algorithms, including state-of-the-art as well as some of the most recent algorithms, for neural network training. In particular, we evaluate particle swarm optimisation (PSO), differential evolution (DE), artificial bee colony (ABC), imperialist competitive algorithm (ICA), cuckoo search (CS), gravitational search algorithm (GSA), bat algorithm (BA), firefly algorithm (FA), grey wolf optimiser (GWO), ant lion optimiser (ALO), dragonfly algorithm (DA), sine cosine algorithm (SCA), whale optimisation algorithm (WOA), grasshopper optimisation algorithm (GOA), and salp swarm algorithm (SSA), and assess their performance on different classification algorithms.

[1]  Adam Slowik,et al.  Application of an Adaptive Differential Evolution Algorithm With Multiple Trial Vectors to Artificial Neural Network Training , 2011, IEEE Transactions on Industrial Electronics.

[2]  Chih-Fong Tsai,et al.  Using neural network ensembles for bankruptcy prediction and credit scoring , 2008, Expert Syst. Appl..

[3]  Xin-She Yang,et al.  Cuckoo Search via Lévy flights , 2009, 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC).

[4]  H. Khanna Nehemiah,et al.  Neural network classifier optimization using Differential Evolution with Global Information and Back Propagation algorithm for clinical datasets , 2016, Appl. Soft Comput..

[5]  Seyed Jalaleddin Mousavirad,et al.  A memetic imperialist competitive algorithm with chaotic maps for multi-layer neural network training , 2019 .

[6]  Shahryar Rahnamayan,et al.  Metaheuristics in large-scale global continues optimization: A survey , 2015, Inf. Sci..

[7]  MirjaliliSeyedali,et al.  Grasshopper Optimisation Algorithm , 2017 .

[8]  Seyed Mohammad Mirjalili,et al.  The Ant Lion Optimizer , 2015, Adv. Eng. Softw..

[9]  Dervis Karaboga,et al.  A novel clustering approach: Artificial Bee Colony (ABC) algorithm , 2011, Appl. Soft Comput..

[10]  Andrew Lewis,et al.  Let a biogeography-based optimizer train your Multi-Layer Perceptron , 2014, Inf. Sci..

[11]  Robert C. Green,et al.  Training neural networks using Central Force Optimization and Particle Swarm Optimization: Insights and comparisons , 2012, Expert Syst. Appl..

[12]  Yue Shi,et al.  A modified particle swarm optimizer , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[13]  Andrew Lewis,et al.  The Whale Optimization Algorithm , 2016, Adv. Eng. Softw..

[14]  Iakov Korovin,et al.  An Effective Hybrid Approach for Optimising the Learning Process of Multi-layer Neural Networks , 2019, ISNN.

[15]  Fardin Akhlaghian Tab,et al.  Classification of Rice Varieties Using Optimal Color and Texture Features and BP Neural Networks , 2011, 2011 7th Iranian Conference on Machine Vision and Image Processing.

[16]  Seyedali Mirjalili,et al.  Dragonfly algorithm: a new meta-heuristic optimization technique for solving single-objective, discrete, and multi-objective problems , 2015, Neural Computing and Applications.

[17]  Seyed Jalaleddin Mousavirad,et al.  A Levy flight-based grey wolf optimizer combined with back-propagation algorithm for neural network training , 2017, Neural Computing and Applications.

[18]  Li Cheng,et al.  A New Metaheuristic Bat-Inspired Algorithm , 2010 .

[19]  Saeid Nahavandi,et al.  An efficient Neuroevolution Approach for Heart Disease Detection , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[20]  Hossein Nezamabadi-pour,et al.  GSA: A Gravitational Search Algorithm , 2009, Inf. Sci..

[21]  Wouter M. Kouw,et al.  CT image segmentation of bone for medical additive manufacturing using a convolutional neural network , 2018, Comput. Biol. Medicine.

[22]  Seyedali Mirjalili,et al.  SCA: A Sine Cosine Algorithm for solving optimization problems , 2016, Knowl. Based Syst..

[23]  Seyed Jalaleddin Mousavirad,et al.  Human mental search: a new population-based metaheuristic optimization algorithm , 2017, Applied Intelligence.

[24]  Xin-She Yang,et al.  Firefly algorithm, stochastic test functions and design optimisation , 2010, Int. J. Bio Inspired Comput..

[25]  Joni-Kristian Kämäräinen,et al.  Differential Evolution Training Algorithm for Feed-Forward Neural Networks , 2003, Neural Processing Letters.

[26]  Hossam Faris,et al.  Salp Swarm Algorithm: A bio-inspired optimizer for engineering design problems , 2017, Adv. Eng. Softw..

[27]  Seyed Mohammad Mirjalili.  How effective is the Grey Wolf optimizer in training multi-layer perceptrons , 2014, Applied Intelligence.

[28]  Andrew Lewis,et al.  Grey Wolf Optimizer , 2014, Adv. Eng. Softw..

[29]  Saeid Nahavandi,et al.  Autonomous Robot Navigation Using Moth-Flame-Based Neuroevolution , 2020 .

[30]  Andrew Lewis,et al.  Grasshopper Optimisation Algorithm: Theory and application , 2017, Adv. Eng. Softw..

[31]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[32]  Caro Lucas,et al.  Imperialist competitive algorithm: An algorithm for optimization inspired by imperialistic competition , 2007, 2007 IEEE Congress on Evolutionary Computation.

[33]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..