Incremental construction of rule ensembles using classifiers produced by different class orderings

In this paper, we discuss a novel approach to incrementally construct a rule ensemble. The approach constructs an ensemble from a dynamically generated set of rule classifiers. Each classifier in this set is trained by using a different class ordering. We investigate criteria including accuracy, ensemble size, and the role of starting point in the search. Fusion is done by averaging. Using 22 data sets, floating search finds small, accurate ensembles in polynomial time.

[1]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[2]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Noel E. Sharkey,et al.  The "Test and Select" Approach to Ensemble Combination , 2000, Multiple Classifier Systems.

[4]  William B. Yates,et al.  Engineering Multiversion Neural-Net Systems , 1996, Neural Computation.

[5]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[6]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[7]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[8]  Zehra Cataltepe,et al.  Extended multimodal Eigenclassifiers and criteria for fusion model selection , 2015, Inf. Sci..

[9]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[10]  Ethem Alpaydin,et al.  Incremental construction of classifier and discriminant ensembles , 2009, Inf. Sci..

[11]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[12]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[13]  Robert Sabourin,et al.  A New Objective Function for Ensemble Selection in Random Subspaces , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[14]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[15]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[16]  Robert P. W. Duin,et al.  An experimental study on diversity for bagging and boosting with linear classifiers , 2002, Inf. Fusion.

[17]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Ludmila I. Kuncheva.  Diversity in multiple classifier systems , 2005, Inf. Fusion.

[19]  João B. D. Cabrera,et al.  On the impact of fusion strategies on classification errors for large ensembles of classifiers , 2006, Pattern Recognit..

[20]  Chun Yang,et al.  Diversity-Based Ensemble with Sample Weight Learning , 2014, 2014 22nd International Conference on Pattern Recognition.

[21]  Tin Kam Ho,et al.  Improving cross-validation based classifier selection using meta-learning , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[22]  Rich Caruana,et al.  Ensemble selection from libraries of models , 2004, ICML.

[23]  Bogdan Gabrys,et al.  Classifier selection for majority voting , 2005, Inf. Fusion.

[24]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[25]  Karthik Ramasubramanian,et al.  Introduction to Machine Learning and R , 2017 .

[26]  Peter Clark,et al.  Rule Induction with CN2: Some Recent Improvements , 1991, EWSL.

[27]  JOHANNES FÜRNKRANZ,et al.  Separate-and-Conquer Rule Learning , 1999, Artificial Intelligence Review.

[28]  Constantin F. Aliferis,et al.  A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis , 2004, Bioinform..

[29]  William Nick Street,et al.  Ensemble Pruning Via Semi-definite Programming , 2006, J. Mach. Learn. Res..

[30]  Ian H. Witten,et al.  Generating Accurate Rule Sets Without Global Optimization , 1998, ICML.

[31]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[32]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[33]  George D. C. Cavalcanti,et al.  On Meta-learning for Dynamic Ensemble Selection , 2014, 2014 22nd International Conference on Pattern Recognition.

[34]  Ethem Alpaydin,et al.  Eigenclassifiers for combining correlated classifiers , 2012, Inf. Sci..

[35]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[36]  Robert Sabourin,et al.  A dynamic overproduce-and-choose strategy for the selection of classifier ensembles , 2008, Pattern Recognit..

[37]  Geoffrey I. Webb,et al.  To Select or To Weigh: A Comparative Study of Linear Combination Schemes for SuperParent-One-Dependence Estimators , 2007, IEEE Transactions on Knowledge and Data Engineering.