The build of n-Bits Binary Coding ICBP Ensemble System

To optimize the construct of an ensemble of many relatively small and simple classifiers may be more realistic than to optimize the design of an individual large and complex classifier. Problems of local minima and slow convergence may be mitigated within an ensemble system where the decisions obtained from locally optimal components classifiers are integrated together. However, it is very difficult to design the structure of individual neural networks (NNs) in an ensemble and the architecture of a whole ensemble. In n-Bits Binary Coding ICBP Ensemble System (nBBC-ICBP-ES), the crucial parameter that is required to be set a priori is an appropriate number of hidden nodes of the corresponding improved circular back-propagation (ICBP) root model. Thereby, both the number of individual ICBPs and the architecture of each ICBP component in an nBBC-ICBP-ES can be decided straightly. nBBC-ICBP-ES is computationally more efficient, with relatively fewer user specified parameters, and does not need any manual division to the training data set for the purpose of its construction. It is rather easy to be understood and implemented, while inheriting the benefits of ICBP root model in a natural manner. Simulation and t-test results on four large-scale benchmark classification data sets demonstrated that, in most cases, nBBC-ICBP-ES significantly improved the classification and generalization performances of the two typically large single ICBPs, Same-2^N^"^h-ICBP-ES (i.e., an ensemble system with the same 2^N^"^h ICBP components), conventional Bagging and AdaBoost ensemble. We conclude that, for NN applications in pattern recognition, assembling many small NNs might be better than just utilizing an individual large one, and further, assembling many heterogeneous small NNs might be better than assembling many homogeneous ones. The proposed nBBC-ICBP-ES is simple but efficient and effective, and also potentially significant for the applications of NN ensemble.

[1]  UedaNaonori Optimal Linear Combination of Neural Networks for Improving Classification Performance , 2000 .

[2]  Yu-Bin Yang,et al.  Lung cancer cell identification based on artificial neural network ensembles , 2002, Artif. Intell. Medicine.

[3]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[4]  Sandro Ridella,et al.  Circular backpropagation networks embed vector quantization , 1999, IEEE Trans. Neural Networks.

[5]  Jeroen Eggermont,et al.  Rule-extraction and learning in the BP-SOM architecture , 1998 .

[6]  Jun Gao,et al.  A survey of neural network ensembles , 2005, 2005 International Conference on Neural Networks and Brain.

[7]  Pádraig Cunningham,et al.  Stability problems with artificial neural networks and the ensemble solution , 2000, Artif. Intell. Medicine.

[8]  Qun Dai,et al.  The build of a dynamic classifier selection ICBP system and its application to pattern recognition , 2010, Neural Computing and Applications.

[9]  Oleksandr Makeyev,et al.  Neural network with ensembles , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[10]  Sandro Ridella,et al.  CBP networks as a generalized neural model , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[11]  Sandro Ridella,et al.  Circular backpropagation networks for classification , 1997, IEEE Trans. Neural Networks.

[12]  Sherif Hashem,et al.  Optimal Linear Combinations of Neural Networks , 1997, Neural Networks.

[13]  Songcan Chen,et al.  Improved CBP Neural Network Model with Applications in Time Series Prediction , 2004, Neural Processing Letters.

[14]  Dai Qun,et al.  Improved CBP Neural Network Model with Applications in Time Series Prediction , 2003 .

[15]  L. Breiman Arcing classifier (with discussion and a rejoinder by the author) , 1998 .

[16]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[17]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[18]  Yoram Singer,et al.  Improved Boosting Algorithms Using Confidence-rated Predictions , 1998, COLT' 98.

[19]  Hou-Kuan Huang,et al.  A selective approach to neural network ensemble based on clustering technology , 2004, Proceedings of 2004 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.04EX826).

[20]  A. J. M. M. Weijters The BP-SOM architecture and learning rule , 2006, Neural Processing Letters.

[21]  R. Schapire The Strength of Weak Learnability , 1990, Machine Learning.

[22]  Li Kai,et al.  A Selective Approach to Neural Network Ensemble Based on Clustering Technology , 2005 .

[23]  Lars Kai Hansen,et al.  Ensemble methods for handwritten digit recognition , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[24]  David W. Opitz,et al.  Actively Searching for an E(cid:11)ective Neural-Network Ensemble , 1996 .

[25]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[26]  H. Jaap van den Herik,et al.  Interpretable Neural Networks with BP-SOM , 1998, ECML.

[27]  Robert L. Winkler,et al.  Limits for the Precision and Value of Information from Dependent Sources , 1985, Oper. Res..

[28]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[29]  L. Breiman Arcing Classifiers , 1998 .

[30]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[31]  Liang Chen,et al.  Classification of 2-dimensional array patterns: Assembling many small neural networks is better than using a large one , 2010, Neural Networks.

[32]  Kevin J. Cherkauer Human Expert-level Performance on a Scientiic Image Analysis Task by a System Using Combined Artiicial Neural Networks , 1996 .

[33]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[34]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[35]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[36]  L. Cooper,et al.  When Networks Disagree: Ensemble Methods for Hybrid Neural Networks , 1992 .

[37]  Shun-ichi Amari,et al.  Mutual information of sparsely coded associative memory with self-control and ternary neurons , 2000, Neural Networks.

[38]  Jude W. Shavlik,et al.  Combining the Predictions of Multiple Classifiers: Using Competitive Learning to Initialize Neural Networks , 1995, IJCAI.

[39]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[40]  Tsuhan Chen,et al.  Pose invariant face recognition , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[41]  H. J. van den Herik,et al.  Intelligible neural networks with BP-SOM , 1997 .

[42]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[43]  William Nick Street,et al.  Ensemble Pruning Via Semi-definite Programming , 2006, J. Mach. Learn. Res..

[44]  William W. Hsieh,et al.  Nonlinear principal component analysis by neural networks , 2001 .

[45]  Xin Yao,et al.  Simultaneous training of negatively correlated neural networks in an ensemble , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[46]  Xin Yao,et al.  A constructive algorithm for training cooperative neural network ensembles , 2003, IEEE Trans. Neural Networks.