Parameter determination and feature selection for back-propagation network by particle swarm optimization

The back-propagation network (BPN) is a popular tool with applications in a variety of fields. Nevertheless, different problems may require different parameter settings for a given network architecture. A dataset may contain many features, but not all features are beneficial for classification by the BPN. Therefore, a particle-swarm-optimization-based approach, denoted as PSOBPN, is proposed to obtain the suitable parameter settings for BPN and to select the beneficial subset of features which result in a better classification accuracy rate. A set of 23 problems with a range of examples and features drawn from the UCI (University of California, Irvine) machine learning repository is adopted to test the performance of the proposed algorithm. The results are compared with several well-known published algorithms. The comparative study shows that the proposed approach improves the classification accuracy rate in most test problems. Furthermore, when the feature selection is taken into consideration, the classification accuracy rates of most datasets are increased. The proposed algorithm should thus be useful to both practitioners and researchers.

[1]  Randall S. Sexton,et al.  Optimization of neural networks: A comparative analysis of the genetic algorithm and simulated annealing , 1999, Eur. J. Oper. Res..

[2]  Juan Julián Merelo Guervós,et al.  G-Prop: Global optimization of multilayer perceptrons using GAs , 2000, Neurocomputing.

[3]  Tai-Yue Wang,et al.  Applying optimized BPN to a chaotic time series problem , 2007, Expert Syst. Appl..

[4]  Chris Clifton,et al.  Database Integration Using Neural Networks: Implementation and Experiences , 2000, Knowledge and Information Systems.

[5]  Michael J. A. Berry,et al.  Data mining techniques - for marketing, sales, and customer support , 1997, Wiley computer publishing.

[6]  Ataollah Ebrahimzadeh,et al.  Intelligent digital signal-type identification , 2008, Eng. Appl. Artif. Intell..

[7]  Wenxin Liu,et al.  Particle swarm optimization-based parameter identification applied to permanent magnet synchronous motors , 2008, Eng. Appl. Artif. Intell..

[8]  Shih-Wei Lin,et al.  Particle swarm optimization for parameter determination and feature selection of support vector machines , 2008, Expert Syst. Appl..

[9]  Yong Liu,et al.  A GA-based NN approach for makespan estimation , 2007, Appl. Math. Comput..

[10]  B. S. Lim,et al.  Optimal design of neural networks using the Taguchi method , 1995, Neurocomputing.

[11]  Yue Shi,et al.  A modified particle swarm optimizer , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[12]  Mineichi Kudo,et al.  Non-parametric classifier-independent feature selection , 2006, Pattern Recognit..

[13]  Kejun Zhu,et al.  A dynamic all parameters adaptive BP neural networks model and its application on oil reservoir prediction , 2008, Appl. Math. Comput..

[14]  Yingtao Jiang,et al.  A multilayer perceptron-based medical decision support system for heart disease diagnosis , 2006, Expert Syst. Appl..

[15]  Il-Seok Oh,et al.  Simultaneous node pruning of input and hidden layers using genetic algorithms , 2008, 2008 International Conference on Machine Learning and Cybernetics.

[16]  Wen-Chin Chen,et al.  Process parameter optimization for MIMO plastic injection molding via soft computing , 2009, Expert Syst. Appl..

[17]  Shih-Chieh Chen,et al.  Irregular shapes classification by back-propagation neural networks , 2007 .

[18]  Randall S. Sexton,et al.  Knowledge discovery using a neural network simultaneous optimization algorithm on a real world classification problem , 2006, Eur. J. Oper. Res..

[19]  Driss Ouazar,et al.  Evolving neural network using real coded genetic algorithm for daily rainfall-runoff forecasting , 2009, Expert Syst. Appl..

[20]  Akbar A. Javadi,et al.  A hybrid intelligent genetic algorithm , 2005, Adv. Eng. Informatics.

[21]  Wen Wang,et al.  Design of neural network-based estimator for tool wear modeling in hard turning , 2008, J. Intell. Manuf..

[22]  Massimiliano Barletta,et al.  Modelling of electrostatic fluidized bed (EFB) coating process using artificial neural networks , 2007, Eng. Appl. Artif. Intell..

[23]  Josef Kittler,et al.  Floating search methods in feature selection , 1994, Pattern Recognit. Lett..

[24]  Shih-Chieh Lin,et al.  Drill wear monitoring using neural networks , 1996 .

[25]  Jihoon Yang,et al.  Feature Subset Selection Using a Genetic Algorithm , 1998, IEEE Intell. Syst..

[26]  Gang Wang,et al.  A Study of Classification Algorithm for Data Mining Based on Hybrid Intelligent Systems , 2008, 2008 Ninth ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing.

[27]  Wai Keung Wong,et al.  A hybrid model using genetic algorithm and neural network for classifying garment defects , 2009, Expert Syst. Appl..

[28]  Wanli Zuo,et al.  SVM based adaptive learning method for text classification from positive and unlabeled documents , 2008, Knowledge and Information Systems.

[29]  Nariman Sepehri,et al.  Neural network classifiers applied to condition monitoring of a pneumatic process valve actuator , 2002 .

[30]  Asoke K. Nandi,et al.  Fault detection using genetic programming , 2005 .

[31]  Ling Wang,et al.  An effective co-evolutionary particle swarm optimization for constrained engineering design problems , 2007, Eng. Appl. Artif. Intell..

[32]  Lakhmi C. Jain,et al.  Evolutionary computation in data mining , 2005 .

[33]  Hiroshi Motoda,et al.  Feature Selection for Knowledge Discovery and Data Mining , 1998, The Springer International Series in Engineering and Computer Science.

[34]  Shuo-Yan Chou,et al.  Location determination of mobile devices for an indoor WLAN application using a neural network , 2009, Knowledge and Information Systems.

[35]  Russell C. Eberhart,et al.  Parameter Selection in Particle Swarm Optimization , 1998, Evolutionary Programming.

[36]  Antanas Verikas,et al.  Feature selection with neural networks , 2002, Pattern Recognit. Lett..

[37]  S. Danaher,et al.  Erosion modelling using Bayesian regulated artificial neural networks , 2004 .

[38]  Ingoo Han,et al.  Genetic algorithms approach to feature discretization in artificial neural networks for the prediction of stock price index , 2000 .

[39]  A. Yamazaki,et al.  Optimization of neural network weights and architectures for odor recognition using simulated annealing , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[40]  Ranadhir Ghosh,et al.  A Hierarchical Method for Finding Optimal Architecture and Weights Using Evolutionary Least Square Based Learning , 2003, Int. J. Neural Syst..

[41]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[42]  Glenn Fung,et al.  Finite Newton method for Lagrangian support vector machine classification , 2003, Neurocomputing.

[43]  Sreeram Ramakrishnan,et al.  A hybrid approach for feature subset selection using neural networks and ant colony optimization , 2007, Expert Syst. Appl..

[44]  Dilip Kumar Pratihar,et al.  Forward and reverse mappings in green sand mould system using neural networks , 2008, Appl. Soft Comput..

[45]  Bahram Alidaee,et al.  Global optimization for artificial neural networks: A tabu search application , 1998, Eur. J. Oper. Res..

[46]  Steven Salzberg,et al.  On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach , 1997, Data Mining and Knowledge Discovery.

[47]  Shuo-Yan Chou,et al.  A simulated-annealing-based approach for simultaneous parameter optimization and feature selection of back-propagation networks , 2008, Expert Syst. Appl..

[48]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[49]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[50]  Takahiro Sasaki,et al.  Comparison between Lamarckian and Darwinian Evolution on a Model Using Neural Networks and Genetic Algorithms , 2000, Knowledge and Information Systems.

[51]  Jongsoo Lee,et al.  GA based meta-modeling of BPN architecture for constrained approximate optimization , 2007 .

[52]  Martin J. Oates,et al.  A Preliminary Investigation of Modified XCS as a Generic Data Mining Tool , 2001, IWLCS.

[53]  Juan Julián Merelo Guervós,et al.  Evolving Multilayer Perceptrons , 2000, Neural Processing Letters.

[54]  Chien-Yu Huang,et al.  Evaluating the process of a genetic algorithm to improve the back-propagation network: A Monte Carlo study , 2009, Expert Syst. Appl..

[55]  Tung-Hsu Hou,et al.  Using neural networks and immune algorithms to find the optimal parameters for an IC wire bonding process , 2008, Expert Syst. Appl..

[56]  Dimitar Filev,et al.  Intelligent systems in the automotive industry: applications and trends , 2007, Knowledge and Information Systems.

[57]  Yong S. Choi,et al.  Discovering Text Databases with Neural Nets , 2001, Knowledge and Information Systems.

[58]  Zhi Gao,et al.  Robust neuro-identification of nonlinear plants in electric power systems with missing sensor measurements , 2008, Eng. Appl. Artif. Intell..

[59]  Randall S. Sexton,et al.  Comparing backpropagation with a genetic algorithm for neural network training , 1999 .

[60]  M. Senthil Arumugam,et al.  A new and improved version of particle swarm optimization algorithm with global–local best parameters , 2008, Knowledge and Information Systems.

[61]  Larry Bull,et al.  Genetic Programming for Kernel-Based Learning with Co-evolving Subsets Selection , 2006, PPSN.

[62]  Larry Bull,et al.  GAP: Constructing and Selecting Features with Evolutionary Computing , 2005 .

[63]  Hubert Cardot,et al.  A Neural Network Architecture for Data Classification , 2001, Int. J. Neural Syst..

[64]  Gamini Dissanayake,et al.  Evolutionary computing based mobile robot localization , 2006, Eng. Appl. Artif. Intell..

[65]  K. Ramanjaneyulu,et al.  An integrated approach for optimum design of bridge decks using genetic algorithms and artificial neural networks , 2007, Adv. Eng. Softw..

[66]  Zne-Jung Lee,et al.  Parameter determination of support vector machine and feature selection using simulated annealing approach , 2008, Appl. Soft Comput..

[67]  Young-Don Ko,et al.  Modeling and optimization of the growth rate for ZnO thin films using neural networks and genetic algorithms , 2009, Expert Syst. Appl..

[68]  David R. Musicant,et al.  Lagrangian Support Vector Machines , 2001, J. Mach. Learn. Res..

[69]  Gustavo Deco,et al.  Two Strategies to Avoid Overfitting in Feedforward Networks , 1997, Neural Networks.

[70]  R. Malhotra,et al.  Evaluating Consumer Loans Using Neural Networks , 2001 .

[71]  R. Suganya,et al.  Data Mining Concepts and Techniques , 2010 .

[72]  D. S. Yeung,et al.  Hidden neuron pruning for multilayer perceptrons using a sensitivity measure , 2002, Proceedings. International Conference on Machine Learning and Cybernetics.