Online Non-linear Gradient Boosting in Multi-latent Spaces

Gradient Boosting is a popular ensemble method that combines linearly diverse and weak hypotheses to build a strong classifier. In this work, we propose a new Online Non-Linear gradient Boosting (ONLB) algorithm where we suggest to jointly learn different combinations of the same set of weak classifiers in order to learn the idiosyncrasies of the target concept. To expand the expressiveness of the final model, our method leverages the non linear complementarity of these combinations. We perform an experimental study showing that ONLB (i) outperforms most recent online boosting methods in both terms of convergence rate and accuracy and (ii) learns diverse and useful new latent spaces.

[1]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[2]  John Langford,et al.  Beating the hold-out: bounds for K-fold and progressive cross-validation , 1999, COLT '99.

[3]  Horst Bischof,et al.  On robustness of on-line boosting - a competitive study , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[4]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[5]  Ya Zhang,et al.  Boosted multi-task learning , 2010, Machine Learning.

[6]  Ambuj Tewari,et al.  Online multiclass boosting , 2017, NIPS.

[7]  Martial Hebert,et al.  Gradient Boosting on Stochastic Data Streams , 2017, AISTATS.

[8]  Stuart J. Russell,et al.  Online bagging and boosting , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[9]  João Gama,et al.  Cascade Generalization , 2000, Machine Learning.

[10]  Horst Bischof,et al.  On-line Boosting and Vision , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[11]  Hsuan-Tien Lin,et al.  An Online Boosting Algorithm with Theoretical Justifications , 2012, ICML.

[12]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[13]  Pascal Fua,et al.  Non-Linear Domain Adaptation with Boosting , 2013, NIPS.

[14]  Haipeng Luo,et al.  Online Gradient Boosting , 2015, NIPS.

[15]  Yan Tong,et al.  Incremental Boosting Convolutional Neural Network for Facial Action Unit Recognition , 2017, NIPS.

[16]  Tie-Yan Liu,et al.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree , 2017, NIPS.

[17]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[18]  Haipeng Luo,et al.  Optimal and Adaptive Algorithms for Online Boosting , 2015, ICML.

[19]  Horst Possegger,et al.  BIER — Boosting Independent Embeddings Robustly , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[20]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[21]  Nicolás García-Pedrajas,et al.  Nonlinear Boosting Projections for Ensemble Construction , 2007, J. Mach. Learn. Res..

[22]  Zhi-Hua Zhou,et al.  One-Pass AUC Optimization , 2013, ICML.

[23]  Ralf Klinkenberg,et al.  Boosting classifiers for drifting concepts , 2007, Intell. Data Anal..