Ensemble selection from libraries of models

We present a method for constructing ensembles from libraries of thousands of models. Model libraries are generated using different learning algorithms and parameter settings. Forward stepwise selection is used to add to the ensemble the models that maximize its performance. Ensemble selection allows ensembles to be optimized to performance metric such as accuracy, cross entropy, mean precision, or ROC Area. Experiments with seven test problems and ten metrics demonstrate the benefit of ensemble selection.

[1]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[2]  Thomas G. Dietterich,et al.  Solving Multiclass Learning Problems via Error-Correcting Output Codes , 1994, J. Artif. Intell. Res..

[3]  Paul W. Munro,et al.  Competition Among Networks Improves Committee Performance , 1996, NIPS.

[4]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[5]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[6]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[7]  David W. Opitz,et al.  Feature Selection for Ensembles , 1999, AAAI/IAAI.

[8]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[9]  Robert F. Cromp,et al.  Support Vector Machine Classifiers as Applied to AVIRIS Data , 1999 .

[10]  Pedro M. Domingos.  Bayesian Averaging of Classifiers and the Overfitting Problem , 2000, ICML.

[11]  Thomas G. Dietterich.  Ensemble Methods in Machine Learning , 2000, Multiple Classifier Systems.

[12]  J. Langford,et al.  FeatureBoost: A Meta-Learning Algorithm that Improves Model Robustness , 2000, ICML.

[13]  Bo Thiesson,et al.  Staged Mixture Modelling and Boosting , 2002, UAI.

[14]  D. Heckerman,et al.  Stage Mixture Modeling and Boosting , 2002 .

[15]  Robert E. Schapire,et al.  The Boosting Approach to Machine Learning An Overview , 2003 .

[16]  Pedro M. Domingos,et al.  Tree Induction for Probability-Based Ranking , 2003, Machine Learning.

[17]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.