Margin-based ordered aggregation for ensemble pruning

Ensemble methods have been successfully used as a classification scheme. The reduction of the complexity of this popular learning paradigm motivated the appearance of ensemble pruning algorithms. This paper presents a new ensemble pruning method which highly reduces the complexity of ensemble methods and performs better than complete bagging in terms of classification accuracy. More importantly, it is a very fast algorithm. It consists in ordering all base classifiers with respect to a new criterion which exploits an unsupervised ensemble margin. This method highlights the major influence of low margin instances on the performance of the pruning task and, more generally, the potential of low margin instances for the design of better ensembles. Comparison to both the naive approach of randomly pruning base classifiers and another ordering-based pruning algorithm is carried out in an extensive empirical analysis.

[1]  Ludmila I. Kuncheva,et al.  Switching between selection and fusion in combining classifiers: an experiment , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[2]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[3]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[4]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[5]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[6]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[7]  Ludmila I. Kuncheva,et al.  Examining the Relationship Between Majority Vote Accuracy and Diversity in Bagging and Boosting , 2003 .

[8]  Samia Boukir,et al.  A two-pass random forests classification of airborne lidar and image data on urban scenes , 2010, 2010 IEEE International Conference on Image Processing.

[9]  Fabio Roli,et al.  Design of effective multiple classifier systems by clustering of classifiers , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[10]  Geoffrey I. Webb,et al.  Ensemble Selection for SuperParent-One-Dependence Estimators , 2005, Australian Conference on Artificial Intelligence.

[11]  Hongyu Zhao,et al.  Building pathway clusters from Random Forests classification using class votes , 2008, BMC Bioinformatics.

[12]  Wei Tang,et al.  Selective Ensemble of Decision Trees , 2003, RSFDGrC.

[13]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[14]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[15]  Daniel Hernández-Lobato,et al.  An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[17]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[18]  Samia Boukir,et al.  Relevance of airborne lidar and multispectral image data for urban scene classification using Random Forests , 2011 .

[19]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[20]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[21]  William B. Yates,et al.  Engineering Multiversion Neural-Net Systems , 1996, Neural Computation.

[22]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Gonzalo Martínez-Muñoz,et al.  Pruning in ordered bagging ensembles , 2006, ICML.

[24]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[25]  Shinichi Morishita,et al.  On Classification and Regression , 1998, Discovery Science.

[26]  Zoran Obradovic,et al.  Effective pruning of neural network classifier ensembles , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[27]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[28]  和司 池田.  INNS-IEEE International Joint Conference on Neural Networks(IJCNN2001)(「人間を内部に含んだ系のモデリングと設計特集号」) , 2001 .

[29]  Tom Heskes,et al.  Clustering ensembles of neural network models , 2003, Neural Networks.

[30]  Samia Boukir,et al.  Support Vectors Selection for Supervised Learning Using an Ensemble Approach , 2010, 2010 20th International Conference on Pattern Recognition.

[31]  William Nick Street,et al.  Ensemble Pruning Via Semi-definite Programming , 2006, J. Mach. Learn. Res..

[32]  Giorgio Valentini,et al.  Applications of Supervised and Unsupervised Ensemble Methods , 2009, Applications of Supervised and Unsupervised Ensemble Methods.

[33]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .