Beam Search Extraction and Forgetting Strategies on Shared Ensembles

Ensemble methods improve accuracy by combining the predictions of a set of different hypotheses. However, there is an important shortcoming associated with ensemble methods. Huge amounts of memory are required to store a set of multiple hypotheses. In this work, we have devised an ensemble method that partially solves this problem. The key point is that components share their common parts. We employ a multi-tree, which is a structure that can simultaneously contain an ensemble of decision trees but has the advantage that decision trees share some conditions. To construct this multi-tree, we define an algorithm based on a beam search with several extraction criteria and with several forgetting policies for the suspended nodes. Finally, we compare the behaviour of this ensemble method with some well-known methods for generating hypothesis ensembles.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[3]  G DietterichThomas.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees , 2000 .

[4]  Ludmila I. Kuncheva,et al.  A Theoretical Study on Six Classifier Fusion Strategies , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  José Hernández-Orallo,et al.  SMILES: A Multi-purpose Learning System , 2002, JELIA.

[6]  Ron Kohavi,et al.  Option Decision Trees with Majority Votes , 1997, ICML.

[7]  Peter Clark,et al.  The CN2 Induction Algorithm , 1989, Machine Learning.

[8]  C. J. Whitaker,et al.  Ten measures of diversity in classifier ensembles: limits for two classifiers , 2001 .

[9]  D. Hand,et al.  Artificial Intelligence Frontiers in Statistics , 2020 .

[10]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[11]  José Hernández-Orallo,et al.  Shared Ensemble Learning Using Multi-trees , 2002, IBERAMIA.

[12]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[13]  Wray L. Buntine,et al.  A theory of learning classification rules , 1990 .

[14]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[15]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[16]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[17]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[18]  Nils J. Nilsson,et al.  Artificial Intelligence: A New Synthesis , 1997 .

[19]  Wray L. Buntine,et al.  Learning classification trees , 1992 .

[20]  Mark S. Boddy,et al.  An Analysis of Time-Dependent Planning , 1988, AAAI.

[21]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.