A new reverse reduce-error ensemble pruning algorithm

An interesting RRE pruning algorithm incorporated with the operation of subtraction is proposed in this work.The WSM is chosen and its votes are subtracted from the votes made by those selected components.The backfitting step of RE algorithm is replaced with the selection step of a WSB in RRE.The problem of ties might be solved more naturally with RRE.Soft voting approach is employed in the testing to RRE algorithm. Although greedy algorithms possess high efficiency, they often receive suboptimal solutions of the ensemble pruning problem, since their exploration areas are limited in large extent. And another marked defect of almost all the currently existing ensemble pruning algorithms, including greedy ones, consists in: they simply abandon all of the classifiers which fail in the competition of ensemble selection, causing a considerable waste of useful resources and information. Inspired by these observations, an interesting greedy Reverse Reduce-Error (RRE) pruning algorithm incorporated with the operation of subtraction is proposed in this work. The RRE algorithm makes the best of the defeated candidate networks in a way that, the Worst Single Model (WSM) is chosen, and then, its votes are subtracted from the votes made by those selected components within the pruned ensemble. The reason is because, for most cases, the WSM might make mistakes in its estimation for the test samples. And, different from the classical RE, the near-optimal solution is produced based on the pruned error of all the available sequential subensembles. Besides, the backfitting step of RE algorithm is replaced with the selection step of a WSM in RRE. Moreover, the problem of ties might be solved more naturally with RRE. Finally, soft voting approach is employed in the testing to RRE algorithm. The performances of RE and RRE algorithms, and two baseline methods, i.e., the method which selects the Best Single Model (BSM) in the initial ensemble, and the method which retains all member networks of the initial ensemble (ALL), are evaluated on seven benchmark classification tasks under different initial ensemble setups. The results of the empirical investigation show the superiority of RRE over the other three ensemble pruning algorithms.

[1]  Jun Gao,et al.  A survey of neural network ensembles , 2005, 2005 International Conference on Neural Networks and Brain.

[2]  Daniel Hernández-Lobato,et al.  An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Pádraig Cunningham,et al.  Stability problems with artificial neural networks and the ensemble solution , 2000, Artif. Intell. Medicine.

[4]  Rich Caruana,et al.  Ensemble selection from libraries of models , 2004, ICML.

[5]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[6]  H. Jaap van den Herik,et al.  Interpretable Neural Networks with BP-SOM , 1998, ECML.

[7]  Lawrence O. Hall,et al.  Ensemble diversity measures and their application to thinning , 2004, Inf. Fusion.

[8]  Qun Dai,et al.  The build of n-Bits Binary Coding ICBP Ensemble System , 2011, Neurocomputing.

[9]  Kuo-Chen Hung,et al.  An efficient fuzzy weighted average algorithm for the military UAV selecting under group decision-making , 2011, Knowl. Based Syst..

[10]  Rong Wang,et al.  Soft-Voting Classification using Locally Linear Reconstruction , 2011, 2011 Seventh International Conference on Computational Intelligence and Security.

[11]  Lefteris Angelis,et al.  Selective fusion of heterogeneous classifiers , 2005, Intell. Data Anal..

[12]  Yu-Bin Yang,et al.  Lung cancer cell identification based on artificial neural network ensembles , 2002, Artif. Intell. Medicine.

[13]  Zhi-Hua Zhou,et al.  Ensemble Methods: Foundations and Algorithms , 2012 .

[14]  Geoffrey I. Webb,et al.  Ensemble Selection for SuperParent-One-Dependence Estimators , 2005, Australian Conference on Artificial Intelligence.

[15]  Philip S. Yu,et al.  Pruning and dynamic scheduling of cost-sensitive ensembles , 2002, AAAI/IAAI.

[16]  Qun Dai,et al.  An efficient ensemble pruning algorithm using One-Path and Two-Trips searching approach , 2013, Knowl. Based Syst..

[17]  Donato Malerba,et al.  A Further Comparison of Simplification Methods for Decision-Tree Induction , 1995, AISTATS.

[18]  Grigorios Tsoumakas,et al.  Effective Voting of Heterogeneous Classifiers , 2004, ECML.

[19]  A. J. M. M. Weijters.  The BP-SOM architecture and learning rule , 2006, Neural Processing Letters.

[20]  Grigorios Tsoumakas,et al.  Pruning an ensemble of classifiers via reinforcement learning , 2009, Neurocomputing.

[21]  William B. Yates,et al.  Engineering Multiversion Neural-Net Systems , 1996, Neural Computation.

[22]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[23]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[24]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[25]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[26]  Qun Dai,et al.  ModEnPBT: A Modified Backtracking Ensemble Pruning algorithm , 2013, Appl. Soft Comput..

[27]  Christino Tamon,et al.  On the Boosting Pruning Problem , 2000, ECML.

[28]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[29]  Tsuhan Chen,et al.  Pose invariant face recognition , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[30]  Łukasz Wróbel,et al.  Application of Rule Induction Algorithms for Analysis of Data Collected by Seismic Hazard Monitoring Systems in Coal Mines , 2010 .

[31]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[32]  Donato Malerba,et al.  Multistrategy Learning for Document Recognition , 1994, Appl. Artif. Intell..

[33]  Grigorios Tsoumakas,et al.  Ensemble Pruning Using Reinforcement Learning , 2006, SETN.

[34]  Dai Qun,et al.  Improved CBP Neural Network Model with Applications in Time Series Prediction , 2003 .

[35]  Tom Heskes,et al.  Clustering ensembles of neural network models , 2003, Neural Networks.

[36]  Grigorios Tsoumakas,et al.  An Ensemble Pruning Primer , 2009, Applications of Supervised and Unsupervised Ensemble Methods.

[37]  William Nick Street,et al.  Ensemble Pruning Via Semi-definite Programming , 2006, J. Mach. Learn. Res..

[38]  S. Karthikeyan,et al.  An ensemble design of intrusion detection system for handling uncertainty using Neutrosophic Logic Classifier , 2012, Knowl. Based Syst..

[39]  Wei Tang,et al.  Selective Ensemble of Decision Trees , 2003, RSFDGrC.

[40]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Grigorios Tsoumakas,et al.  An ensemble uncertainty aware measure for directed hill climbing ensemble pruning , 2010, Machine Learning.

[42]  Fabio Roli,et al.  Design of effective multiple classifier systems by clustering of classifiers , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[43]  Fu Qiang,et al.  Clustering-based selective neural network ensemble , 2005 .

[44]  B. Roe,et al.  Boosted decision trees as an alternative to artificial neural networks for particle identification , 2004, physics/0408124.

[45]  Qun Dai,et al.  A competitive ensemble pruning approach based on cross-validation technique , 2013, Knowl. Based Syst..

[46]  Lars Kai Hansen,et al.  Ensemble methods for handwritten digit recognition , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[47]  Qun Dai,et al.  A novel ensemble pruning algorithm based on randomized greedy selective strategy and ballot , 2013, Neurocomputing.

[48]  Gonzalo Martínez-Muñoz,et al.  Using boosting to prune bagging ensembles , 2007, Pattern Recognit. Lett..

[49]  Gonzalo Martínez-Muñoz,et al.  Pruning in ordered bagging ensembles , 2006, ICML.