A Short Introduction to Boosting

Boosting is a general method for improving the accuracy of any given learning algorithm. This short overview paper introduces the boosting algorithm AdaBoost, and explains the underlying theory of boosting, including an explanation of why boosting often does not suffer from overfitting as well as boosting’s relationship to support-vector machines. Some examples of recent applications of boosting are also described.

[1]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[2]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[3]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[4]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[5]  Yoav Freund,et al.  Boosting a weak learning algorithm by majority , 1995, COLT '90.

[6]  Yoav Freund,et al.  Boosting a weak learning algorithm by majority , 1990, COLT '90.

[7]  Robert E. Schapire,et al.  The strength of weak learnability , 1990, Mach. Learn..

[8]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[9]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[10]  Harris Drucker,et al.  Boosting Performance in Neural Networks , 1993, Int. J. Pattern Recognit. Artif. Intell..

[11]  Leslie G. Valiant,et al.  Cryptographic Limitations on Learning Boolean Formulae and Finite Automata , 1993, Machine Learning: From Theory to Applications.

[12]  Leslie G. Valiant,et al.  Cryptographic limitations on learning Boolean formulae and finite automata , 1994, JACM.

[13]  Umesh V. Vazirani,et al.  An Introduction to Computational Learning Theory , 1994 .

[14]  Johannes Fürnkranz,et al.  Incremental Reduced Error Pruning , 1994, ICML.

[15]  Mark Craven,et al.  Learning Sparse Perceptrons , 1995, NIPS.

[16]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[17]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[18]  金田 重郎,et al.  C4.5: Programs for Machine Learning (書評) , 1995 .

[19]  Corinna Cortes,et al.  Boosting Decision Trees , 1995, NIPS.

[20]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[21]  Thomas G. Dietterich,et al.  Solving Multiclass Learning Problems via Error-Correcting Output Codes , 1994, J. Artif. Intell. Res..

[22]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[23]  J. Ross Quinlan,et al.  Bagging, Boosting, and C4.5 , 1996, AAAI/IAAI, Vol. 1.

[24]  Yoav Freund,et al.  Game theory, on-line prediction and boosting , 1996, COLT '96.

[25]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[26]  Yoshua Bengio,et al.  Training Methods for Adaptive Boosting of Neural Networks , 1997, NIPS.

[27]  David W. Opitz,et al.  An Empirical Evaluation of Bagging and Boosting , 1997, AAAI/IAAI.

[28]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[29]  L. Breiman.  Arcing the edge , 1997 .

[30]  Robert E. Schapire,et al.  Using output codes to boost multiclass learning problems , 1997, ICML.

[31]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[32]  Peter L. Bartlett,et al.  Direct Optimization of Margins Improves Generalization in Combined Classifiers , 1998, NIPS.

[33]  Dale Schuurmans,et al.  Boosting in the Limit: Maximizing the Margin of Learned Ensembles , 1998, AAAI/IAAI.

[34]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[35]  Using Decision Trees to Construct a Practical Parser , 1998, COLING-ACL.

[36]  L. Breiman.  Arcing Classifiers , 1998 .

[37]  Yoram Singer,et al.  Improved Boosting Algorithms Using Confidence-rated Predictions , 1998, COLT' 98.

[38]  L. Breiman.  Arcing classifier (with discussion and a rejoinder by the author) , 1998 .

[39]  R. Tibshirani,et al.  Additive Logistic Regression : a Statistical View ofBoostingJerome , 1998 .

[40]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[41]  Yoram Singer,et al.  Boosting and Rocchio applied to text filtering , 1998, SIGIR '98.

[42]  Y. Freund,et al.  Adaptive game playing using multiplicative weights , 1999 .

[43]  Yoav Freund,et al.  The Alternating Decision Tree Learning Algorithm , 1999, ICML.

[44]  Robert E. Schapire,et al.  Drifting Games , 1999, Annual Conference Computational Learning Theory.

[45]  Yoram Singer,et al.  Boosting Applied to Tagging and PP Attachment , 1999, EMNLP.

[46]  Yoram Singer,et al.  A simple, fast, and effective rule learner , 1999, AAAI 1999.

[47]  Yoav Freund,et al.  An Adaptive Version of the Boost by Majority Algorithm , 1999, COLT.

[48]  J. Friedman.  Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[49]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.