Adaptive Splitting and Selection Algorithm for Regression

Abstract Developing system for regression tasks like predicting prices, temperature is not a trivial task. There are many of issues which must be addressed such as: selecting appropriate model, eliminating irrelevant inputs, removing noise, etc. Most of them can be solved by application of machine learning methods. Although most of them were developed for classification tasks, they can be successfully applied for regression too. Therefore, in this paper we present Adaptive Splitting and Selection for Regression algorithm, whose predecessor was successfully applied in many classification tasks. The algorithm uses ensemble techniques whose strength comes from exploring local competences of several predictors. This is achieved by decomposing input space into disjointed competence areas and establishing local ensembles for each area respectively. Learning procedure is implemented as a compound optimisation process solved by means of evolutionary algorithm. The performance of the system is evaluated in series of experiments carried on several benchmark datasets. Obtained results show that proposed algorithm is valuable option for those who look for regression method.

[1]  Michal Prilepok,et al.  Evolutionary Weighted Ensemble for EEG Signal Recognition , 2014, ECC.

[2]  Wolfgang Härdle,et al.  Applied Nonparametric Regression , 1991 .

[3]  Fernando José Von Zuben,et al.  The Influence of the Pool of Candidates on the Performance of Selection and Combination Techniques in Ensembles , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[4]  Max A. Little,et al.  Accurate Telemonitoring of Parkinson's Disease Progression by Noninvasive Speech Tests , 2009, IEEE Transactions on Biomedical Engineering.

[5]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[6]  Gonzalo Mart,et al.  Pruning in Ordered Regression Bagging Ensembles , 2006 .

[7]  Jesús Alcalá-Fdez,et al.  KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework , 2011, J. Multiple Valued Log. Soft Comput..

[8]  Harris Drucker,et al.  Improving Regressors using Boosting Techniques , 1997, ICML.

[9]  Gerardo Rubino,et al.  Echo State Queueing Network: A new reservoir computing learning tool , 2012, 2013 IEEE 10th Consumer Communications and Networking Conference (CCNC).

[10]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[11]  Ethem Alpaydın,et al.  Combined 5 x 2 cv F Test for Comparing Supervised Classification Learning Algorithms , 1999, Neural Comput..

[12]  Michal Wozniak,et al.  Adaptive Splitting and Selection Method of Classifier Ensemble Building , 2009, HAIS.

[13]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[14]  Bartosz Krawczyk,et al.  APPLICATION OF ADAPTIVE SPLITTING AND SELECTION CLASSIFIER TO THE SPAM FILTERING PROBLEM , 2013, Cybern. Syst..

[15]  Pınar Tüfekci,et al.  Prediction of full load electrical power output of a base load operated combined cycle power plant using machine learning methods , 2014 .

[16]  Daniel Hernández-Lobato,et al.  Empirical analysis and evaluation of approximate techniques for pruning regression bagging ensembles , 2011, Neurocomputing.

[17]  Yong Wang,et al.  Modeling for Optimal Probability Prediction , 2002, ICML.

[18]  Zbigniew Telec,et al.  A Multi-Agent System to Assist with Property Valuation Using Heterogeneous Ensembles of Fuzzy Models , 2010, KES-AMSTA.

[19]  Durga L. Shrestha,et al.  Experiments with AdaBoost.RT, an Improved Boosting Scheme for Regression , 2006, Neural Computation.

[20]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[21]  Peter Tiño,et al.  Managing Diversity in Regression Ensembles , 2005, J. Mach. Learn. Res..

[22]  S. Salzberg,et al.  A weighted nearest neighbor algorithm for learning with symbolic features , 2004, Machine Learning.

[23]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[24]  Stephen Tyree,et al.  Parallel boosted regression trees for web search ranking , 2011, WWW.

[25]  I-Cheng Yeh,et al.  Modeling of strength of high-performance concrete using artificial neural networks , 1998 .

[26]  Francisco Herrera,et al.  Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power , 2010, Inf. Sci..

[27]  Hubert Cardot,et al.  A new boosting algorithm for improved time-series forecasting with recurrent neural networks , 2008, Inf. Fusion.

[28]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[29]  Peter J. Rousseeuw,et al.  Robust Regression and Outlier Detection , 2005, Wiley Series in Probability and Statistics.

[30]  Nathan Intrator,et al.  Boosting Regression Estimators , 1999, Neural Computation.

[31]  Gerardo Rubino,et al.  Levenberg - Marquardt Training Algorithms for Random Neural Networks , 2011, Comput. J..

[32]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[33]  Ethem Alpaydin,et al.  Introduction to machine learning , 2004, Adaptive computation and machine learning.