A Study on the Noise Label Influence in Boosting Algorithms: AdaBoost, GBM and XGBoost

In classification, class noise alludes to incorrect labelling of instances and it causes the classifiers to perform worse. In this contribution, we test the resistance against noise of the most influential boosting algorithms. We explain the fundamentals of these state-of-the-art algorithms, providing an unified notation to facilitate their comparison. We analyse how they carry out the classification, what loss functions use and what techniques employ under the boosting scheme.

[1]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[2]  J. Friedman.  Stochastic gradient boosting , 2002 .

[3]  Francisco Herrera,et al.  Tutorial on practical tips of the most influential data preprocessing algorithms in data mining , 2016, Knowl. Based Syst..

[4]  Bo Sun,et al.  A robust multi-class AdaBoost algorithm for mislabeled noisy data , 2016, Knowl. Based Syst..

[5]  Matthias Templ,et al.  Estimation of Social Exclusion Indicators from Complex Surveys: The R Package Laeken , 2012 .

[6]  Francisco Herrera,et al.  Evaluating the classifier behavior with noisy data considering performance and robustness: The Equalized Loss of Accuracy measure , 2016, Neurocomputing.

[7]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[8]  Francisco Herrera,et al.  Data Preprocessing in Data Mining , 2014, Intelligent Systems Reference Library.

[9]  Gunnar Rätsch,et al.  Soft Margins for AdaBoost , 2001, Machine Learning.

[10]  Stephen Kwek,et al.  A boosting approach to remove class label noise , 2005, Fifth International Conference on Hybrid Intelligent Systems (HIS'05).

[11]  Greg Ridgeway,et al.  Generalized Boosted Models: A guide to the gbm package , 2006 .

[12]  Francisco Herrera,et al.  A First Study on the Use of Boosting for Class Noise Reparation , 2016, HAIS.

[13]  Sam Kwong,et al.  A noise-detection based AdaBoost algorithm for mislabeled data , 2012, Pattern Recognit..

[14]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[15]  David J. Hand,et al.  An Empirical Comparison of Three Boosting Algorithms on Real Data Sets with Artificial Class Noise , 2003, Multiple Classifier Systems.

[16]  Matías Gámez,et al.  adabag: An R Package for Classification with Boosting and Bagging , 2013 .

[17]  Maoguo Gong,et al.  RBoost: Label Noise-Robust Boosting Algorithm Based on a Nonconvex Loss Function and the Numerically Stable Base Learners , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[18]  J. Skilling.  Bayesian Methods in Cosmology: Foundations and algorithms , 2009 .

[19]  M. Verleysen,et al.  Classification in the Presence of Label Noise: A Survey , 2014, IEEE Transactions on Neural Networks and Learning Systems.