Designing Artificial Neural Networks Using Particle Swarm Optimization Algorithms

Artificial Neural Network (ANN) design is a complex task because its performance depends on the architecture, the selected transfer function, and the learning algorithm used to train the set of synaptic weights. In this paper we present a methodology that automatically designs an ANN using particle swarm optimization algorithms such as Basic Particle Swarm Optimization (PSO), Second Generation of Particle Swarm Optimization (SGPSO), and a New Model of PSO called NMPSO. The aim of these algorithms is to evolve, at the same time, the three principal components of an ANN: the set of synaptic weights, the connections or architecture, and the transfer functions for each neuron. Eight different fitness functions were proposed to evaluate the fitness of each solution and find the best design. These functions are based on the mean square error (MSE) and the classification error (CER) and implement a strategy to avoid overtraining and to reduce the number of connections in the ANN. In addition, the ANN designed with the proposed methodology is compared with those designed manually using the well-known Back-Propagation and Levenberg-Marquardt Learning Algorithms. Finally, the accuracy of the method is tested with different nonlinear pattern classification problems.

[1]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[2]  Juan Humberto Sossa Azuela,et al.  Evolving Neural Networks: A Comparison between Differential Evolution and Particle Swarm Optimization , 2011, ICSI.

[3]  Hitoshi Iba,et al.  Particle swarm optimization with Gaussian mutation , 2003, Proceedings of the 2003 IEEE Swarm Intelligence Symposium. SIS'03 (Cat. No.03EX706).

[4]  Juan Humberto Sossa Azuela,et al.  A New Associative Model with Dynamical Synapses , 2008, Neural Processing Letters.

[5]  Christian Posthoff,et al.  Earthquake classifying neural networks trained with random dynamic neighborhood PSOs , 2007, GECCO '07.

[6]  Ge Xiurun,et al.  An improved PSO-based ANN with simulated annealing technique , 2005, Neurocomputing.

[7]  Sobri Harun,et al.  Particle swarm optimization feedforward neural network for modeling runoff , 2010 .

[8]  R. Eberhart,et al.  Empirical study of particle swarm optimization , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[9]  Daniel Rivero,et al.  Evolving Graphs for ANN Development and Simplification , 2009, Encyclopedia of Artificial Intelligence.

[10]  Yan Meng,et al.  Toward Evolving Neural Networks using Bio-Inspired Algorithms , 2008, IC-AI.

[11]  Mingquan Chen,et al.  Second Generation Particle Swarm Optimization , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[12]  Lifeng Xi,et al.  An Improved Particle Swarm Optimization for Evolving Feedforward Artificial Neural Networks , 2007, Neural Processing Letters.

[13]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[14]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[15]  Juan Humberto Sossa Azuela,et al.  Design of artificial neural networks using a modified Particle Swarm Optimization algorithm , 2009, 2009 International Joint Conference on Neural Networks.

[16]  Giulio Sandini,et al.  Robots and Biological Systems: Towards a New Bionics? , 2012, NATO ASI Series.

[17]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[18]  Thomas Kiel Rasmussen,et al.  Hybrid Particle Swarm Optimiser with breeding and subpopulations , 2001 .

[19]  H. M. Abdul,et al.  Neural Networks Training Based on Differential Evolution Algorithm Compared with Other Architectures for Weather Forecasting34 , 2009 .

[20]  Jing Wang,et al.  Swarm Intelligence in Cellular Robotic Systems , 1993 .

[21]  Juan Humberto Sossa Azuela,et al.  Design of Artificial Neural Networks Using Differential Evolution Algorithm , 2010, ICONIP.

[22]  Juan Humberto Sossa Azuela,et al.  Artificial neural network synthesis by means of artificial bee colony (ABC) algorithm , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[23]  Patrick van der Smagt,et al.  Introduction to neural networks , 1995, The Lancet.

[24]  Enrique Alba,et al.  Metaheuristic Procedures for Training Neural Networks (Operations Research/Computer Science Interfaces Series) , 2006 .

[25]  Beatriz A. Garro,et al.  Back-Propagation vs Particle Swarm Optimization Algorithm: which Algorithm is better to adjust the Synaptic Weights of a Feed-Forward ANN? , 2011 .