An Adaptive Memetic Algorithm With Rank-Based Mutation for Artificial Neural Network Architecture Optimization

Designing a well-generalized architecture for artificial neural networks (ANNs) is an important task. This paper presents an adaptive memetic algorithm with a rank-based mutation, denoted as AMARM, to design ANN architectures. The proposed algorithm introduces an adaptive multi-local search mechanism to simultaneously fine-tune the number of hidden neurons and connection weights. The adaptation of the multi-local search mechanism is achieved by identifying effective local searches based on their search characteristics. Such an algorithm is distinguishable from previous evolutionary algorithm-based methods that incorporate one single local search for evolving ANN architectures. Furthermore, a rank-based mutation strategy is devised for avoiding premature convergence during evolution. The performance of the proposed algorithm has been evaluated on a number of benchmark problems and compared with related work. The results show that the AMARM can be used to design compact ANN architectures with good generalization capability, outperforming related work.

[1]  Jihoon Yang,et al.  Constructive Neural-Network Learning Algorithms for Pattern Classification , 2000 .

[2]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[3]  Christian Blum,et al.  An ant colony optimization algorithm for continuous optimization: application to feed-forward neural network training , 2007, Neural Computing and Applications.

[4]  Hak-Keung Lam,et al.  Tuning of the structure and parameters of a neural network using an improved genetic algorithm , 2003, IEEE Trans. Neural Networks.

[5]  Gan Xuehui,et al.  Condition Monitoring and Faults Recognizing of Dish Centrifugal Separator by Artifical Neural Network Combined with Expert System , 2009, 2009 Fifth International Conference on Natural Computation.

[6]  Lutz Prechelt,et al.  A quantitative study of experimental evaluations of neural network learning algorithms: Current research practice , 1996, Neural Networks.

[7]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[8]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[9]  Xin Yao,et al.  A constructive algorithm for training cooperative neural network ensembles , 2003, IEEE Trans. Neural Networks.

[10]  Michael J. Dinneen,et al.  A (1+1) Adaptive Memetic Algorithm for the Maximum Clique Problem , 2013, 2013 IEEE Congress on Evolutionary Computation.

[11]  Chee Peng Lim,et al.  A Modified Fuzzy Min–Max Neural Network With a Genetic-Algorithm-Based Rule Extractor for Pattern Classification , 2010, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[12]  P. D. Heermann,et al.  Classification of multispectral remote sensing data using a back-propagation neural network , 1992, IEEE Trans. Geosci. Remote. Sens..

[13]  Lifeng Xi,et al.  Evolving artificial neural networks using an improved PSO and DPSO , 2008, Neurocomputing.

[14]  Teresa Bernarda Ludermir,et al.  A multi-objective memetic and hybrid methodology for optimizing the parameters and performance of artificial neural networks , 2010, Neurocomputing.

[15]  Alfred Jean Philippe Lauret,et al.  A node pruning algorithm based on a Fourier amplitude sensitivity test method , 2006, IEEE Transactions on Neural Networks.

[16]  Sung-Kwun Oh,et al.  Self-organizing neurofuzzy networks based on evolutionary fuzzy granulation , 2003, IEEE Trans. Syst. Man Cybern. Part A.

[17]  Peng Shi,et al.  Adaptive Neural Fault-Tolerant Control of a 3-DOF Model Helicopter System , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[18]  Madhumita Panda,et al.  A Hybrid Differential Evolution and Back-Propagation Algorithm for Feedforward Neural Network Training , 2013 .

[19]  A. C. Martínez-Estudillo,et al.  Hybridization of evolutionary algorithms and local search by means of a clustering method , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[20]  Xiaojian Ma,et al.  Condition Monitoring and Faults Recognizing of Dish Centrifugal Separator by Artifical Neural Network Combined with Expert System , 2009, ICNC.

[21]  Andy J. Keane,et al.  Meta-Lamarckian learning in memetic algorithms , 2004, IEEE Transactions on Evolutionary Computation.

[22]  Xin Yao,et al.  Evolutionary programming using mutations based on the Levy probability distribution , 2004, IEEE Transactions on Evolutionary Computation.

[23]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[24]  Weiguo Sheng,et al.  An Adaptive Memetic Algorithm for Designing Artificial Neural Networks , 2015, 2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom).

[25]  César Hervás-Martínez,et al.  COVNET: a cooperative coevolutionary model for evolving artificial neural networks , 2003, IEEE Trans. Neural Networks.

[26]  William E. Hart,et al.  Memetic Evolutionary Algorithms , 2005 .

[27]  Chia-Feng Juang,et al.  A hybrid of genetic algorithm and particle swarm optimization for recurrent network design , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[28]  Hitoshi Iba,et al.  Learning polynomial feedforward neural networks by genetic programming and backpropagation , 2003, IEEE Trans. Neural Networks.

[29]  Bingxue Shi,et al.  Hybrid BP-GA for multilayer feedforward neural networks , 2000, ICECS 2000. 7th IEEE International Conference on Electronics, Circuits and Systems (Cat. No.00EX445).

[30]  H. Altun,et al.  Treatment of multi-dimensional data to enhance neural network estimators in regression problems , 2006 .

[31]  Kevin Kok Wai Wong,et al.  Classification of adaptive memetic algorithms: a comparative study , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[32]  Alberto Tesi,et al.  On the Problem of Local Minima in Backpropagation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Chin-Teng Lin,et al.  An Improved Polynomial Neural Network Classifier Using Real-Coded Genetic Algorithm , 2015, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[34]  Per Kristian Lehre,et al.  Theoretical analysis of rank-based mutation - combining exploration and exploitation , 2009, 2009 IEEE Congress on Evolutionary Computation.

[35]  Andries Petrus Engelbrecht,et al.  A new pruning heuristic based on variance analysis of sensitivity information , 2001, IEEE Trans. Neural Networks.

[36]  Gary G. Yen,et al.  Rank-density-based multiobjective genetic algorithm and benchmark test function study , 2003, IEEE Trans. Evol. Comput..

[37]  MengChu Zhou,et al.  Improved Quantum-Inspired Evolutionary Algorithm for Large-Size Lane Reservation , 2015, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[38]  Yew-Soon Ong,et al.  Artificial intelligence technologies in complex engineering design , 2002 .

[39]  Lalit M. Patnaik,et al.  Adaptive probabilities of crossover and mutation in genetic algorithms , 1994, IEEE Trans. Syst. Man Cybern..

[40]  Kay Chen Tan,et al.  A Multi-Facet Survey on Memetic Computation , 2011, IEEE Transactions on Evolutionary Computation.

[41]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[42]  Shiro Usui,et al.  Mutation-based genetic neural network , 2005, IEEE Transactions on Neural Networks.

[43]  A. P. Wieland,et al.  Evolving neural network controllers for unstable systems , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[44]  Mark Sumner,et al.  A Fast Adaptive Memetic Algorithm for Online and Offline Control Design of PMSM Drives , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[45]  Ying-Ping Chen,et al.  Analysis on the Collaboration Between Global Search and Local Search in Memetic Computation , 2011, IEEE Transactions on Evolutionary Computation.

[46]  Derek B. Ingham,et al.  Fitness Diversity Based Adaptive Memetic Algorithm for solving inverse problems of chemical kinetics , 2007, 2007 IEEE Congress on Evolutionary Computation.

[47]  Weiguo Sheng,et al.  Multilocal Search and Adaptive Niching Based Memetic Algorithm With a Consensus Criterion for Data Clustering , 2014, IEEE Transactions on Evolutionary Computation.

[48]  Masoud Yaghini,et al.  A hybrid algorithm for artificial neural network training , 2013, Eng. Appl. Artif. Intell..

[49]  Leonardo Franco,et al.  C-Mantec: A novel constructive neural network algorithm incorporating competition between neurons , 2012, Neural Networks.

[50]  Kazuyuki Murase,et al.  A New Constructive Algorithm for Designing and Training Artificial Neural Networks , 2007, ICONIP.

[51]  Nor Ashidi Mat Isa,et al.  Adaptive Evolutionary Artificial Neural Networks for Pattern Classification , 2011, IEEE Transactions on Neural Networks.

[52]  Tung-Kuan Liu,et al.  Tuning the structure and parameters of a neural network by using hybrid Taguchi-genetic algorithm , 2006, IEEE Trans. Neural Networks.

[53]  E. Cantu-Paz,et al.  An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[54]  Shih-Hung Yang,et al.  An evolutionary constructive and pruning algorithm for artificial neural networks and its prediction applications , 2012, Neurocomputing.

[55]  Xin Yao,et al.  A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[56]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[57]  Derong Liu,et al.  Data-Based Adaptive Critic Designs for Nonlinear Robust Optimal Control With Uncertain Dynamics , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[58]  Changyin Sun,et al.  Neural Network Control of a Robotic Manipulator With Input Deadzone and Output Constraint , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[59]  Achintya Das,et al.  Training a Feed-forward Neural Network with Artificial Bee Colony Based Backpropagation Method , 2012, ArXiv.

[60]  Beatriz A. Garro,et al.  Designing Artificial Neural Networks Using Particle Swarm Optimization Algorithms , 2015, Comput. Intell. Neurosci..

[61]  Pratyusha Rakshit,et al.  Realization of an Adaptive Memetic Algorithm Using Differential Evolution and Q-Learning: A Case Study in Multirobot Path Planning , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[62]  Hong Li,et al.  ODE-LM: A Hybrid Training Algorithm for Feedforward Neural Networks , 2014 .

[63]  Mengjie Zhang,et al.  Crossover-based local search in cooperative co-evolutionary feedforward neural networks , 2012, Appl. Soft Comput..

[64]  Xin Yao,et al.  A New Constructive Algorithm for Architectural and Functional Adaptation of Artificial Neural Networks , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[65]  Teresa Bernarda Ludermir,et al.  Hybrid Training Method for MLP: Optimization of Architecture and Training , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).