Extremal Optimization Combined with LM Gradient Search for MLP Network Learning

Gradient search based neural network training algorithm may suffer from local optimum, poor generalization and slow convergence. In this study, a novel Memetic Algorithm based hybrid method with the integration of “extremal optimization” and “Levenberg–Marquardt” is proposed to train multilayer perceptron (MLP) networks. Inheriting the advantages of the two approaches, the proposed “EO-LM” method can avoid local minima and improve MLP network learning performance in generalization capability and computation efficiency. The experimental tests on two benchmark problems and an application example for the end-point-prediction of basic oxygen furnace in steelmaking show the effectiveness of the proposed EO-LM algorithm.

[1]  Jin-Kao Hao,et al.  A memetic algorithm for graph coloring , 2010, Eur. J. Oper. Res..

[2]  Wout Dullaert,et al.  A penalty-based edge assembly memetic algorithm for the vehicle routing problem with time windows , 2010, Comput. Oper. Res..

[3]  Jacek M. Zurada,et al.  Identification of Full and Partial Class Relevant Genes , 2010, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[4]  Robert W. Harrison,et al.  Granular Decision Tree and Evolutionary Neural SVM for Protein Secondary Structure Prediction , 2009, Int. J. Comput. Intell. Syst..

[5]  Kyandoghere Kyamakya,et al.  Cellular Neural Networks-Based Genetic Algorithm for Optimizing the Behavior of an Unstructured Robot , 2009, Int. J. Comput. Intell. Syst..

[6]  Driss Ouazar,et al.  Evolving neural network using real coded genetic algorithm for daily rainfall-runoff forecasting , 2009, Expert Syst. Appl..

[7]  Orhan Dengiz,et al.  A tabu search algorithm for the training of neural networks , 2009, J. Oper. Res. Soc..

[8]  Min-Rong Chen,et al.  A novel elitist multiobjective optimization algorithm: Multiobjective extremal optimization , 2008, Eur. J. Oper. Res..

[9]  Yu-Wang Chen,et al.  Hybrid evolutionary algorithm with marriage of genetic algorithm and extremal optimization for production scheduling , 2008 .

[10]  Xin Yao,et al.  Evolving artificial neural network ensembles , 2008, IEEE Computational Intelligence Magazine.

[11]  Peng Chen,et al.  Optimization with extremal dynamics for the traveling salesman problem , 2007 .

[12]  Min-Rong Chen,et al.  Studies on Extremal Optimization and Its Applications in Solving RealWorld Optimization Problems , 2007, 2007 IEEE Symposium on Foundations of Computational Intelligence.

[13]  Yong Liu,et al.  A GA-based NN approach for makespan estimation , 2007, Appl. Math. Comput..

[14]  Ender Özcan,et al.  Memes, Self-generation and Nurse Rostering , 2006, PATAT.

[15]  Tim Hendtlass,et al.  Solving Problems with Hidden Dynamics – Comparison of Extremal Optimisation and Ant Colony System , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[16]  B. Foss,et al.  A new optimization algorithm with application to nonlinear MPC , 2004 .

[17]  Fabiano Luis de Sousa,et al.  Generalized extremal optimization: An application in heat pipe design , 2004 .

[18]  S. Boettcher.  Extremal optimization for Sherrington-Kirkpatrick spin glasses , 2004, cond-mat/0407130.

[19]  Stefan Boettcher,et al.  Extremal Optimization for the Sherrington-Kirkpatrick Spin Glass , 2004 .

[20]  Natalio Krasnogor,et al.  A Study on the use of ``self-generation'' in memetic algorithms , 2004, Natural Computing.

[21]  A. Percus,et al.  Extremal optimization at the phase transition of the three-coloring problem. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[22]  Mohamed El Bachir Menai,et al.  Efficient Initial Solution to Extremal Optimization Algorithm for Weighted MAXSAT Problem , 2003, IEA/AIE.

[23]  Jasmina Arifovic,et al.  Using genetic algorithms to select architecture of a feedforward artificial neural network , 2001 .

[24]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[25]  Don R. Hush,et al.  Training a Sigmoidal Node Is Hard , 1999, Neural Computation.

[26]  A. Percus,et al.  Extremal Optimization: Methods derived from Co-Evolution , 1999, GECCO.

[27]  A. Percus,et al.  Nature's Way of Optimizing , 1999, Artif. Intell..

[28]  Yutaka Fukuoka,et al.  A modified back-propagation method to avoid false local minima , 1998, Neural Networks.

[29]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[30]  Ralf Salomon,et al.  Evolutionary algorithms and gradient search: similarities and differences , 1998, IEEE Trans. Evol. Comput..

[31]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[32]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[33]  Bak,et al.  Punctuated equilibrium and criticality in a simple model of evolution. , 1993, Physical review letters.

[34]  Thomas P. Vogl,et al.  Rescaling of variables in back propagation learning , 1991, Neural Networks.

[35]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[36]  Tang,et al.  Self-Organized Criticality: An Explanation of 1/f Noise , 2011 .

[37]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[38]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[39]  R. Lewontin.  ‘The Selfish Gene’ , 1977, Nature.

[40]  Pablo Moscato,et al.  Memetic Algorithms , 2007, Handbook of Approximation Algorithms and Metaheuristics.

[41]  Andries P. Engelbrecht,et al.  Computational Intelligence: An Introduction , 2002 .

[42]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[43]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .

[44]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .