An Improved Hybrid Algorithm Based on PSO and BP for Feedforward Neural Networks

In this paper, an improved hybrid algorithm combining particle swarm optimization (PSO) with backpropagation algorithm (BP) is proposed to train feedforward neural networks (FNN). PSO is a global search algorithm, but the swarm in PSO is easy to lose its diversity, which results in premature convergence. On the other hand, BP algorithm is a gradient-descent-based method which has good local search ability around the global minima. Hence, the presented algorithm in this study combines PSO with BP to perform double search. Moreover, in order to improve the diversity of the swarm in the PSO, each particle in the swarm and its best position are disturbed by a random function, and the best position of all particles are reset as the optimum weights of FNN obtained by BP. The proposed algorithm improves the diversity of the swarm as well as reduces the likelihood of the swarm being trapped into local minima on the error surface. Compared with the traditional learning algorithms, the improved learning algorithm has much better convergence accuracy and rate. Finally, the experimental results are given to verify the efficiency and effectiveness of the proposed algorithm.

[1]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[2]  D. E. Goldberg,et al.  Genetic Algorithms in Search, Optimization & Machine Learning , 1989 .

[3]  Andries Petrus Engelbrecht,et al.  Fundamentals of Computational Swarm Intelligence , 2005 .

[4]  Mauro Birattari,et al.  Swarm Intelligence , 2012, Lecture Notes in Computer Science.

[5]  J R Saunders,et al.  A particle swarm optimizer with passive congregation. , 2004, Bio Systems.

[6]  Stephen A. Billings,et al.  Lattice Dynamical Wavelet Neural Networks Implemented Using Particle Swarm Optimization for Spatio–Temporal System Identification , 2009, IEEE Transactions on Neural Networks.

[7]  Moncef Gabbouj,et al.  Evolutionary artificial neural networks by multi-dimensional particle swarm optimization , 2009, Neural Networks.

[8]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[9]  S. Y. Yuen,et al.  A Genetic Algorithm That Adaptively Mutates and Never Revisits , 2009, IEEE Transactions on Evolutionary Computation.

[10]  R. Eberhart,et al.  Empirical study of particle swarm optimization , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[11]  De-Shuang Huang,et al.  Modified constrained learning algorithms incorporating additional functional constraints into neural networks , 2008, Inf. Sci..

[12]  Yuhui Shi,et al.  Particle swarm optimization: developments, applications and resources , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[13]  P. Radhakrishnan,et al.  Extensive Analysis and Prediction of Optimal Inventory levels in supply chain management based on Particle Swarm Optimization Algorithm , 2009, J. Convergence Inf. Technol..

[14]  Ling Wang,et al.  An effective hybrid PSOSA strategy for optimization and its application to parameter estimation , 2006, Appl. Math. Comput..

[15]  Dezhao Chen,et al.  An improved differential evolution algorithm in training and encoding prior knowledge into feedforward networks with application in chemistry , 2002 .

[16]  Jiangye Yuan,et al.  A modified particle swarm optimizer with dynamic adaptation , 2007, Appl. Math. Comput..

[17]  Jacques Riget,et al.  A Diversity-Guided Particle Swarm Optimizer - the ARPSO , 2002 .

[18]  Guanzheng Tan,et al.  Conditional Sensor Deployment Using Evolutionary Algorithms , 2010, J. Convergence Inf. Technol..

[19]  Michael R. Lyu,et al.  A hybrid particle swarm optimization-back-propagation algorithm for feedforward neural network training , 2007, Appl. Math. Comput..

[20]  Fei Han,et al.  A new approach for function approximation incorporating adaptive particle swarm optimization and a priori information , 2008, Appl. Math. Comput..

[21]  Cheng-Jian Lin,et al.  A self-adaptive quantum radial basis function network for classification applications , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[22]  Michael R. Lyu,et al.  A hybrid particle swarm optimization-back-propagation algorithm for feedforward neural network training , 2007, Appl. Math. Comput..

[23]  W. Kinsner,et al.  Chaotic simulated annealing in multilayer feedforward networks , 1996, Proceedings of 1996 Canadian Conference on Electrical and Computer Engineering.

[24]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[25]  Shiu Yin Yuen,et al.  A Genetic Algorithm That Adaptively Mutates and Never Revisits , 2009, IEEE Trans. Evol. Comput..

[26]  Xiaodong Li,et al.  This article has been accepted for inclusion in a future issue. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 1 Locating and Tracking Multiple Dynamic Optima by a Particle Swarm Model Using Speciation , 2022 .

[27]  Sun Zhiyi,et al.  Application of combined neural networks in nonlinear function approximation , 2000, Proceedings of the 3rd World Congress on Intelligent Control and Automation (Cat. No.00EX393).

[28]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[29]  Yi Hong,et al.  An improved particle swarm optimization based training algorithm for neural network , 2005, ICMIT: Mechatronics and Information Technology.

[30]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.