An Ensemble Pruning Primer

Ensemble pruning deals with the reduction of an ensemble of predictive models in order to improve its efficiency and predictive performance. The last 12 years a large number of ensemble pruning methods have been proposed. This work proposes a taxonomy for their organization and reviews important representative methods of each category. It abstracts their key components and discusses their main advantages and disadvantages. We hope that this work will serve as a good starting point and reference for researchers working on the development of new ensemble pruning methods.

[1]  Geoffrey I. Webb,et al.  Ensemble Selection for SuperParent-One-Dependence Estimators , 2005, Australian Conference on Artificial Intelligence.

[2]  Grigorios Tsoumakas,et al.  Ensemble Pruning Using Reinforcement Learning , 2006, SETN.

[3]  Shichao Zhang,et al.  AI 2005: Advances in Artificial Intelligence, 18th Australian Joint Conference on Artificial Intelligence, Sydney, Australia, December 5-9, 2005, Proceedings , 2005, Australian Conference on Artificial Intelligence.

[4]  Zoran Obradovic,et al.  Effective pruning of neural network classifier ensembles , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[5]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[6]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[7]  Dino Pedreschi,et al.  Machine Learning: ECML 2004 , 2004, Lecture Notes in Computer Science.

[8]  Grigorios Tsoumakas,et al.  Pruning an ensemble of classifiers via reinforcement learning , 2009, Neurocomputing.

[9]  Geoffrey I. Webb,et al.  To Select or To Weigh: A Comparative Study of Linear Combination Schemes for SuperParent-One-Dependence Estimators , 2007, IEEE Transactions on Knowledge and Data Engineering.

[10]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[11]  William B. Yates,et al.  Engineering Multiversion Neural-Net Systems , 1996, Neural Computation.

[12]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[13]  Fabio Roli,et al.  Design of effective multiple classifier systems by clustering of classifiers , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[14]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[15]  A. Tamhane,et al.  Multiple Comparison Procedures , 2009 .

[16]  Gonzalo Martínez-Muñoz,et al.  Pruning in ordered bagging ensembles , 2006, ICML.

[17]  Tom Heskes,et al.  Clustering ensembles of neural network models , 2003, Neural Networks.

[18]  Xin Yao,et al.  An analysis of diversity measures , 2006, Machine Learning.

[19]  Fu Qiang,et al.  Clustering-based selective neural network ensemble , 2005 .

[20]  Gonzalo Martínez-Muñoz,et al.  Using boosting to prune bagging ensembles , 2007, Pattern Recognit. Lett..

[21]  Robert E. Schapire,et al.  The strength of weak learnability , 1990, Mach. Learn..

[22]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[23]  Philip S. Yu,et al.  Pruning and dynamic scheduling of cost-sensitive ensembles , 2002, AAAI/IAAI.

[24]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[25]  J. Hsu.  Constrained Simultaneous Confidence Intervals for Multiple Comparisons with the Best , 1984 .

[26]  Grigorios Tsoumakas,et al.  Effective Voting of Heterogeneous Classifiers , 2004, ECML.

[27]  傅强,et al.  Clustering-based selective neural network ensemble , 2005 .

[28]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[29]  Daniel Hernández-Lobato,et al.  An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  A. Tamhane,et al.  Multiple Comparison Procedures , 1989 .

[31]  A. Scott,et al.  A Cluster Analysis Method for Grouping Means in the Analysis of Variance , 1974 .

[32]  Rich Caruana,et al.  Ensemble selection from libraries of models , 2004, ICML.

[33]  Lawrence O. Hall,et al.  Ensemble diversity measures and their application to thinning , 2004, Inf. Fusion.

[34]  Lefteris Angelis,et al.  Selective fusion of heterogeneous classifiers , 2005, Intell. Data Anal..

[35]  Grigorios Tsoumakas,et al.  Focused Ensemble Selection: A Diversity-Based Method for Greedy Ensemble Selection , 2008, ECAI.

[36]  Alberto Suárez,et al.  Aggregation Ordering in Bagging , 2004 .

[37]  William Nick Street,et al.  Ensemble Pruning Via Semi-definite Programming , 2006, J. Mach. Learn. Res..

[38]  Thomas G. Dietterich.  Machine-Learning Research Four Current Directions , 1997 .

[39]  J. Neher.  A problem of multiple comparisons , 2011 .

[40]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[41]  Wei Tang,et al.  Selective Ensemble of Decision Trees , 2003, RSFDGrC.