An evolutionary constructive and pruning algorithm for artificial neural networks and its prediction applications

We propose a method for designing artificial neural networks (ANNs) for prediction problems based on an evolutionary constructive and pruning algorithm (ECPA). The proposed ECPA begins with a set of ANNs with the simplest possible structure, one hidden neuron connected to an input node, and employs crossover and mutation operators to increase the complexity of an ANN population. Additionally, cluster-based pruning (CBP) and age-based survival selection (ABSS) are proposed as two new operators for ANN pruning. The CBP operator retains significant neurons and prunes insignificant neurons on a probability basis and therefore prevents the exponential growth of an ANN. The ABSS operator can delete old ANNs with potentially complex structures and then introduce new ANNs with simple structures; thus, the ANNs are less likely to be trapped in a fully connected topology. The ECPA framework incorporates constructive and pruning approaches in an attempt to efficiently evolve compact ANNs. As a demonstration of the method, ECPA is applied to three prediction problems: the Mackey-Glass time series, the number of sunspots, and traffic flow. The numerical results show that ECPA makes the design of ANNs more feasible and practical for real-world applications.

[1]  David C. Yu,et al.  Weather sensitive short-term load forecasting using nonfully connected artificial neural network , 1992 .

[2]  Kwang Bo Cho,et al.  Radial basis function based adaptive fuzzy systems and their applications to system identification and prediction , 1996, Fuzzy Sets Syst..

[3]  David A. Elizondo,et al.  A Survey of Partially Connected Neural Networks , 1997, Int. J. Neural Syst..

[4]  Guoqiang Peter Zhang,et al.  Time series forecasting using a hybrid ARIMA and neural network model , 2003, Neurocomputing.

[5]  M. Xia,et al.  Adaptive neural network model for time-series forecasting , 2010, Eur. J. Oper. Res..

[6]  Michael Y. Hu,et al.  Forecasting with artificial neural networks: The state of the art , 1997 .

[7]  Andrew Canning,et al.  Partially connected models of neural networks , 1988 .

[8]  Jiwen Dong,et al.  Time-series prediction using a local linear wavelet neural network , 2006, Neurocomputing.

[9]  Teresa Bernarda Ludermir,et al.  Hybrid Training Method for MLP: Optimization of Architecture and Training , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[10]  J. Deng,et al.  Introduction to Grey system theory , 1989 .

[11]  Hak-Keung Lam,et al.  Tuning of the structure and parameters of a neural network using an improved genetic algorithm , 2003, IEEE Trans. Neural Networks.

[12]  César Hervás-Martínez,et al.  Multilogistic regression by means of evolutionary product-unit neural networks , 2008, Neural Networks.

[13]  Shih-Hung Yang,et al.  Intelligent Forecasting System Using Grey Model Combined with Neural Network , 2011 .

[14]  Xin Yao,et al.  A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[15]  Héctor Pomares,et al.  Time series analysis using normalized PG-RBF network with regression weights , 2002, Neurocomputing.

[16]  Bart Kosko,et al.  Neural networks and fuzzy systems: a dynamical systems approach to machine intelligence , 1991 .

[17]  Yoshio Hirose,et al.  Backpropagation algorithm which varies the number of hidden units , 1989, International 1989 Joint Conference on Neural Networks.

[18]  Andries Petrus Engelbrecht,et al.  A new pruning heuristic based on variance analysis of sensitivity information , 2001, IEEE Trans. Neural Networks.

[19]  Shiro Usui,et al.  Mutation-based genetic neural network , 2005, IEEE Transactions on Neural Networks.

[20]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[21]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[22]  Léon Personnaz,et al.  Neural-network construction and selection in nonlinear modeling , 2003, IEEE Trans. Neural Networks.

[23]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[24]  Jacek M. Zurada,et al.  Sensitivity analysis for minimization of input data dimension for feedforward neural network , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[25]  Jacek M. Zurada,et al.  Perturbation method for deleting redundant inputs of perceptron networks , 1997, Neurocomputing.

[26]  De-Shuang Huang,et al.  A Constructive Hybrid Structure Optimization Methodology for Radial Basis Probabilistic Neural Networks , 2008, IEEE Transactions on Neural Networks.

[27]  Kay Chen Tan,et al.  Hybrid Multiobjective Evolutionary Design for Artificial Neural Networks , 2008, IEEE Transactions on Neural Networks.

[28]  Haiping Du,et al.  Time series prediction using evolving radial basis function networks with new encoding scheme , 2008, Neurocomputing.

[29]  M. Holschneider,et al.  Adaptive metrics in the nearest neighbours method , 2008 .

[30]  Akter Hussain,et al.  Price Prediction of Share Market using Artificial Neural Network (ANN) , 2011 .

[31]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[32]  Bruce Curry,et al.  Seasonality and neural networks: a new approach , 2010, Int. J. Metaheuristics.

[33]  Teresa Bernarda Ludermir,et al.  An Optimization Methodology for Neural Network Weights and Architectures , 2006, IEEE Transactions on Neural Networks.

[34]  John W. Tukey,et al.  Exploratory Data Analysis. , 1979 .

[35]  Jyh-Shing Roger Jang,et al.  ANFIS: adaptive-network-based fuzzy inference system , 1993, IEEE Trans. Syst. Man Cybern..

[36]  Mansooreh Mollaghasemi,et al.  AG-ART: An adaptive approach to evolving ART architectures , 2009, Neurocomputing.

[37]  J. Makhoul,et al.  Linear prediction: A tutorial review , 1975, Proceedings of the IEEE.

[38]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[39]  John E. Markel,et al.  Linear Prediction of Speech , 1976, Communication and Cybernetics.

[40]  Christian W. Dawson,et al.  The effect of different basis functions on a radial basis function network for time series prediction: A comparative study , 2006, Neurocomputing.

[41]  Fred Collopy,et al.  How effective are neural networks at forecasting and prediction? A review and evaluation , 1998 .

[42]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[43]  Sanggil Kang,et al.  Partially connected feedforward neural networks structured by input types , 2005, IEEE Transactions on Neural Networks.

[44]  Claudio De Stefano,et al.  On the role of population size and niche radius in fitness sharing , 2004, IEEE Transactions on Evolutionary Computation.

[45]  George C. Anastassopoulos,et al.  Genetic algorithm pruning of probabilistic neural networks in medical disease estimation , 2011, Neural Networks.

[46]  Leandro Nunes de Castro,et al.  Neuro-immune approach to solve routing problems , 2009, Neurocomputing.