Searching Parsimonious Solutions with GA-PARSIMONY and XGBoost in High-Dimensional Databases

EXtreme Gradient Boosting (XGBoost) has become one of the most successful techniques in machine learning competitions. It is computationally efficient and scalable, it supports a wide variety of objective functions and it includes different mechanisms to avoid over-fitting and improve accuracy. Having so many tuning parameters, soft computing (SC) is an alternative to search precise and robust models against classical hyper-tuning methods. In this context, we present a preliminary study in which a SC methodology, named GA-PARSIMONY, is used to find accurate and parsimonious XGBoost solutions. The methodology was designed to optimize the search of parsimonious models by feature selection, parameter tuning and model selection. In this work, different experiments are conducted with four complexity metrics in six high dimensional datasets. Although XGBoost performs well with high-dimensional databases, preliminary results indicated that GA-PARSIMONY with feature selection slightly improved the testing error. Therefore, the choice of solutions with fewer inputs, between those with similar cross-validation errors, can help to obtain more robust solutions with better generalization capabilities.

[1]  Rajkumar Roy,et al.  Evolutionary computing in manufacturing industry: an overview of recent applications , 2005, Appl. Soft Comput..

[2]  Andrés Sanz-García,et al.  Parsimonious Support Vector Machines Modelling for Set Points in Industrial Processes Based on Genetic Algorithm Optimization , 2013, SOCO-CISIS-ICEUTE.

[3]  Witold Jacak,et al.  Analysis of Selected Evolutionary Algorithms in Feature Selection and Parameter Optimization for Data Based Tumor Marker Modeling , 2011, EUROCAST.

[4]  Richard J. Duro,et al.  Evolutionary algorithm characterization in real parameter optimization problems , 2013, Appl. Soft Comput..

[5]  Priyanka,et al.  Genetic algorithms tuned expert model for detection of epileptic seizures from EEG signatures , 2014, Appl. Soft Comput..

[6]  F. Wilcoxon Individual Comparisons by Ranking Methods , 1945 .

[7]  J. Shaffer Modified Sequentially Rejective Multiple Test Procedures , 1986 .

[8]  Julio Fern'andez-Ceniceros,et al.  Methodology based on genetic optimisation to develop overall parsimony models for predicting temperature settings on annealing furnace , 2014 .

[9]  Jianming Ye On Measuring and Correcting the Effects of Data Mining and Model Selection , 1998 .

[10]  Giovanni Seni,et al.  Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions , 2010, Ensemble Methods in Data Mining.

[11]  V. Sadasivam,et al.  An integrated PSO for parameter determination and feature selection of ELM and its application in classification of power system disturbances , 2015, Appl. Soft Comput..

[12]  Julio Fern'andez-Ceniceros,et al.  A numerical-informational approach for characterising the ductile behaviour of the T-stub component. Part 2: Parsimonious soft-computing-based metamodel , 2015 .

[13]  Sheng Ding,et al.  Spectral and Wavelet-based Feature Selection with Particle Swarm Optimization for Hyperspectral Classification , 2011, J. Softw..

[14]  Zbigniew Michalewicz,et al.  Handling Constraints in Genetic Algorithms , 1991, ICGA.

[15]  Emilio Corchado,et al.  Recent trends in intelligent data analysis , 2014, Neurocomputing.

[16]  Hui-Ling Huang,et al.  ESVM: Evolutionary support vector machine for automatic feature selection and classification of microarray data , 2007, Biosyst..

[17]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[18]  F. J. Martinez-de-Pison,et al.  Generation of daily global solar irradiation with support vector machines for regression , 2015 .

[19]  Rubén Urraca,et al.  Improving Hotel Room Demand Forecasting with a Hybrid GA-SVR Methodology Based on Skewed Data Transformation, Feature Selection and Parsimony Tuning , 2015, HAIS.

[20]  Andrés Sanz-García,et al.  GA-PARSIMONY: A GA-SVR approach with feature selection and parameter optimization to obtain parsimonious solutions for predicting temperature settings in a continuous annealing furnace , 2015, Appl. Soft Comput..

[21]  J. Friedman Greedy function approximation: A gradient boosting machine. , 2001 .

[22]  Ning Chen,et al.  A genetic algorithm-based approach to cost-sensitive bankruptcy prediction , 2011, Expert Syst. Appl..

[23]  Mengjie Zhang,et al.  Particle swarm optimisation for feature selection in classification: Novel initialisation and updating mechanisms , 2014, Appl. Soft Comput..

[24]  Andreas Dengel,et al.  Meta-learning for evolutionary parameter optimization of classifiers , 2012, Machine Learning.