Analysis of Generalization Ability for Different AdaBoost Variants Based on Classification and Regression Trees

As a machine learning method, Ada Boost is widely applied to data classification and object detection because of its robustness and efficiency. AdaBoost constructs a global and optimal combination ofweak classifiers based on a sample reweighting. It is known that this kind of combination improves the classification performance tremendously. As the popularity of AdaBoost increases, many variants have been proposed to improve the performance of AdaBoost. Then, a lot of comparison and review studies for AdaBoost variants have also been published. Some researchers compared different AdaBoost variants by experiments in their own fields, and others reviewed various AdaBoost variants by basically introducing these algorithms. However, there is a lack of mathematical analysis of the generalization abilities for different AdaBoost variants. In this paper, we analyze the generalization abilities of six AdaBoost variants in terms of classification margins. The six compared variants are Real AdaBoost, Gentle AdaBoost, Modest AdaBoost, Parameterized AdaBoost, Margin-pruning Boost, and Penalized AdaBoost. Finally, we use experiments to verify our analyses.

[1]  Rocco A. Servedio,et al.  Smooth boosting and learning with malicious noise , 2003 .

[2]  Yi Liu,et al.  SemiBoost: Boosting for Semi-Supervised Learning , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[4]  Dongbing Gu,et al.  Empirical analysis of AdaBoost algorithms on license plate detection , 2009, 2009 International Conference on Mechatronics and Automation.

[5]  Stan Z. Li,et al.  FloatBoost learning and statistical face detection , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Kin-Man Lam,et al.  Feature subset selection for efficient AdaBoost training , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[7]  FU Zhong-liang.  Analysis and Improvement on Real AdaBoost Algorithm , 2012 .

[8]  Vipin Kumar,et al.  Evaluating boosting algorithms to classify rare classes: comparison and improvements , 2001, Proceedings 2001 IEEE International Conference on Data Mining.

[9]  Yang Wang,et al.  Cost-sensitive boosting for classification of imbalanced data , 2007, Pattern Recognit..

[10]  Ying Wang,et al.  The Performance Comparison of Adaboost and SVM Applied to SAR ATR , 2006, 2006 CIE International Conference on Radar.

[11]  Ayhan Demiriz,et al.  Linear Programming Boosting via Column Generation , 2002, Machine Learning.

[12]  Che-Hao Chang,et al.  Improved Hand Tracking System , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[14]  Etienne Grossmann.  AdaTree: Boosting a Weak Classifier into a Decision Tree , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[15]  Yoram Singer,et al.  Improved Boosting Algorithms Using Confidence-rated Predictions , 1998, COLT' 98.

[16]  David A. Elizondo,et al.  Bankruptcy forecasting: An empirical comparison of AdaBoost and neural networks , 2008, Decis. Support Syst..

[17]  Yanfeng Zhang,et al.  A revised AdaBoost algorithm: FM-AdaBoost , 2010, International Conference on Computer Application and System Modeling.

[18]  Wolfgang Förstner,et al.  Comparison of Adaboost and ADTboost for Feature Subset Selection , 2018, PRIS.

[19]  Tolga Tasdizen,et al.  Fast AdaBoost training using weighted novelty selection , 2011, The 2011 International Joint Conference on Neural Networks.

[20]  Yoav Freund,et al.  An Adaptive Version of the Boost by Majority Algorithm , 1999, COLT.

[21]  Joachim Denzler,et al.  Performance Comparison and Evaluation of AdaBoost and SoftBoost Algorithms on Generic Object Recognition , 2008 .

[22]  Y. Freund,et al.  Discussion of the Paper \additive Logistic Regression: a Statistical View of Boosting" By , 2000 .

[23]  M. Narasimha Murty,et al.  Speeding up AdaBoost Classifier with Random Projection , 2009, 2009 Seventh International Conference on Advances in Pattern Recognition.

[24]  Qi Tian,et al.  Interactive Boosting for Image Classification , 2007, MCS.

[25]  Yoav Freund,et al.  A more robust boosting algorithm , 2009, 0905.2138.

[26]  Mathias Kölsch,et al.  Robust hand detection , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[27]  Che-Hao Chang,et al.  Complexity Reduced Face Detection Using Probability-Based Face Mask Prefiltering and Pixel-Based Hierarchical-Feature Adaboosting , 2011, IEEE Signal Processing Letters.

[28]  Yoav Freund,et al.  A Short Introduction to Boosting , 1999 .

[29]  James M. Ferryman,et al.  Faster learning via optimised Adaboost , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[30]  Hiroshi Nagahashi,et al.  Parameterized AdaBoost: Introducing a Parameter to Speed Up the Training of Real AdaBoost , 2014, IEEE Signal Processing Letters.

[31]  F. Huang,et al.  Breast cancer survivability via AdaBoost algorithms , 2008 .

[32]  Ludmila I. Kuncheva.  Error Bounds for Aggressive and Conservative AdaBoost , 2003, Multiple Classifier Systems.

[33]  Kai Ming Ting,et al.  A Comparative Study of Cost-Sensitive Boosting Algorithms , 2000, ICML.

[34]  Osamu Watanabe,et al.  MadaBoost: A Modification of AdaBoost , 2000, COLT.

[35]  Ivan Petrovic,et al.  Experimental comparison of AdaBoost algorithms applied on leg detection with different range sensor setups , 2010, 19th International Workshop on Robotics in Alpe-Adria-Danube Region (RAAD 2010).

[36]  Juan José Rodríguez Diez,et al.  Boosting recombined weak classifiers , 2008, Pattern Recognit. Lett..

[37]  Joseph K. Bradley,et al.  FilterBoost: Regression and Classification on Large Datasets , 2007, NIPS.

[38]  Mário A. T. Figueiredo,et al.  Boosting Algorithms: A Review of Methods, Theory, and Applications , 2012 .

[39]  Xin Miao,et al.  A comparison of random forest and Adaboost tree in ecosystem classification in east Mojave Desert , 2010, 2010 18th International Conference on Geoinformatics.

[40]  Gunnar Rätsch,et al.  Soft Margins for AdaBoost , 2001, Machine Learning.

[41]  Hiroshi Nagahashi,et al.  Penalized AdaBoost: Improving the Generalization Error of Gentle AdaBoost through a Margin Distribution , 2015, IEICE Trans. Inf. Syst..

[42]  Qiang Wu,et al.  Fast and Accurate Human Detection Using a Cascade of Boosted MS-LBP Features , 2012, IEEE Signal Processing Letters.

[43]  Chunhua Shen,et al.  Boosting the Minimum Margin: LPBoost vs. AdaBoost , 2008, 2008 Digital Image Computing: Techniques and Applications.

[44]  Jian Li,et al.  Two new regularized AdaBoost algorithms , 2004, 2004 International Conference on Machine Learning and Applications, 2004. Proceedings..

[45]  Alexander Vezhnevets,et al.  ‘ Modest AdaBoost ’ – Teaching AdaBoost to Generalize Better , 2005 .

[46]  Ke Chen,et al.  Semi-Supervised Learning via Regularized Boosting Working on Multiple Semi-Supervised Assumptions , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Salvatore J. Stolfo,et al.  AdaCost: Misclassification Cost-Sensitive Boosting , 1999, ICML.

[48]  Gunnar Rätsch,et al.  Boosting Algorithms for Maximizing the Soft Margin , 2007, NIPS.

[49]  S. V. N. Vishwanathan,et al.  Entropy Regularized LPBoost , 2008, ALT.

[50]  Taghi M. Khoshgoftaar,et al.  Resampling or Reweighting: A Comparison of Boosting Implementations , 2008, 2008 20th IEEE International Conference on Tools with Artificial Intelligence.

[51]  Hiroshi Nagahashi,et al.  A new method for solving overfitting problem of gentle AdaBoost , 2014, International Conference on Graphic and Image Processing.