Boosting strategy for classification

This paper introduces a strategy for training ensemble classifiers by analysing boosting within margin theory. We present a bound on the generalisation error of ensembled classifiers in terms of the 2-norm of the margin slack vector. We develop an effective, adaptive and robust boosting algorithm, DMBoost, by optimising this bound. The soft margin based quadratic loss function is insensitive to points having a large margin. The algorithm improves the generalisation performance of a system by ignoring the examples having small or negative margin. We evaluate the efficacy of the proposed method by applying it to a text categorization task. Experimental results show that DMBoost performs significantly better than AdaBoost, hence validating the effectiveness of the method. Furthermore, experimental results on UCI data sets demonstrate that DMBoost generally outperforms AdaBoost.

[1]  Yoshua Bengio,et al.  Training Methods for Adaptive Boosting of Neural Networks , 1997, NIPS.

[2]  Dale Schuurmans,et al.  Boosting in the Limit: Maximizing the Margin of Learned Ensembles , 1998, AAAI/IAAI.

[3]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[4]  David W. Opitz,et al.  An Empirical Evaluation of Bagging and Boosting , 1997, AAAI/IAAI.

[5]  J. Ross Quinlan,et al.  Bagging, Boosting, and C4.5 , 1996, AAAI/IAAI, Vol. 1.

[6]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[7]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[8]  Yoram Singer,et al.  BoosTexter: A Boosting-based System for Text Categorization , 2000, Machine Learning.

[9]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[10]  Susan T. Dumais,et al.  Inductive learning algorithms and representations for text categorization , 1998, CIKM '98.

[11]  G DietterichThomas.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees , 2000 .

[12]  Gunnar Rätsch,et al.  Soft Margins for AdaBoost , 2001, Machine Learning.

[13]  John Shawe-Taylor,et al.  A Column Generation Algorithm For Boosting , 2000, ICML.

[14]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[15]  Yoram Singer,et al.  Boosting and Rocchio applied to text filtering , 1998, SIGIR '98.

[16]  Tong Zhang.  Analysis of Regularized Linear Functions for Classification Problems , 1999 .

[17]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[18]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[19]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[20]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[21]  Nello Cristianini,et al.  Further results on the margin distribution , 1999, COLT '99.

[22]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[23]  Corinna Cortes,et al.  Boosting Decision Trees , 1995, NIPS.