Ensemble Learning: A Study on Different Variants of the Dynamic Selection Approach

Integration methods for ensemble learning can use two different approaches: combination or selection. The combination approach (also called fusion) consists on the combination of the predictions obtained by different models in the ensemble to obtain the final ensemble prediction. The selection approach selects one (or more) models from the ensemble according to the prediction performance of these models on similar data from the validation set. Usually, the method to select similar data is the k-nearest neighbors with the Euclidean distance. In this paper we discuss other approaches to obtain similar data for the regression problem. We show that using similarity measures according to the target values improves results. We also show that selecting dynamically several models for the prediction task increases prediction accuracy comparing to the selection of just one model.

[1]  Rich Caruana,et al.  Ensemble selection from libraries of models , 2004, ICML.

[2]  Ludmila I. Kuncheva,et al.  Switching between selection and fusion in combining classifiers: an experiment , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[3]  Eamonn J. Keogh,et al.  Ensembles of Nearest Neighbor Forecasts , 2006, ECML.

[4]  Jan Komorowski,et al.  Principles of Data Mining and Knowledge Discovery , 2001, Lecture Notes in Computer Science.

[5]  César Hervás-Martínez,et al.  Cooperative coevolution of artificial neural network ensembles for pattern classification , 2005, IEEE Transactions on Evolutionary Computation.

[6]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[7]  Robert Tibshirani,et al.  Discriminant Adaptive Nearest Neighbor Classification , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Sunil Arya,et al.  An optimal algorithm for approximate nearest neighbor searching fixed dimensions , 1998, JACM.

[9]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[10]  Luca Didaci,et al.  Dynamic Classifier Selection by Adaptive k-Nearest-Neighbourhood Rule , 2004, Multiple Classifier Systems.

[11]  Michael J. Pazzani,et al.  Classification and regression by combining models , 1998 .

[12]  Fabio Roli,et al.  Dynamic Classifier Selection , 2000, Multiple Classifier Systems.

[13]  Shusaku Tsumoto,et al.  Foundations of Intelligent Systems, 15th International Symposium, ISMIS 2005, Saratoga Springs, NY, USA, May 25-28, 2005, Proceedings , 2005, ISMIS.

[14]  H. Lilliefors.  On the Kolmogorov-Smirnov Test for Normality with Mean and Variance Unknown , 1967 .

[15]  Nojun Kwak,et al.  Feature extraction for classification problems and its application to face recognition , 2008, Pattern Recognit..

[16]  Alexey Tsymbal,et al.  Bagging and Boosting with Dynamic Integration of Classifiers , 2000, PKDD.

[17]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[18]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[19]  Jon Louis Bentley,et al.  Multidimensional binary search trees used for associative searching , 1975, CACM.

[20]  Mykola Pechenizkiy,et al.  Dynamic Integration with Random Forests , 2006, ECML.

[21]  Alexey Tsymbal,et al.  A Dynamic Integration Algorithm for an Ensemble of Classifiers , 1999, ISMIS.

[22]  Teresa Bernarda Ludermir,et al.  Meta-learning approaches to selecting time series models , 2004, Neurocomputing.

[23]  Fabio Roli,et al.  Adaptive Selection of Image Classifiers , 1997, ICIAP.

[24]  Robert Sabourin,et al.  From dynamic classifier selection to dynamic ensemble selection , 2008, Pattern Recognit..

[25]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[26]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[27]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[28]  Marko Robnik-Sikonja,et al.  Theoretical and Empirical Analysis of ReliefF and RReliefF , 2003, Machine Learning.

[29]  Alexey Tsymbal,et al.  Dynamic Integration of Regression Models , 2004, Multiple Classifier Systems.

[30]  Stefano Levialdi,et al.  Image Analysis and Processing , 1987 .

[31]  Kevin W. Bowyer,et al.  Combination of Multiple Classifiers Using Local Accuracy Estimates , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[33]  Christopher J. Merz,et al.  Dynamical Selection of Learning Algorithms , 1995, AISTATS.

[34]  Neil D. Lawrence,et al.  Missing Data in Kernel PCA , 2006, ECML.