Random gradient boosting for predicting conditional quantiles

Gradient Boosting (GB) was introduced to address both classification and regression problems with great power. People have studied the boosting with L2 loss intensively both in theory and practice. However, the L2 loss is not proper for learning distributional functionals beyond the conditional mean such as conditional quantiles. There are huge amount of literatures studying conditional quantile prediction with various methods including machine learning techniques such like random forests and boosting. Simulation studies reveal that the weakness of random forests lies in predicting centre quantiles and that of GB lies in predicting extremes. Is there an algorithm that enjoys the advantages of both random forests and boosting so that it can perform well over all quantiles? In this article, we propose such a boosting algorithm called random GB which embraces the merits of both random forests and GB. Empirical results will be presented to support the superiority of this algorithm in predicting conditional quantiles.

[1]  J. Friedman Greedy function approximation: A gradient boosting machine. , 2001 .

[2]  B. Cade,et al.  Estimating effects of limiting factors with regression quantiles , 1999 .

[3]  Nicolai Meinshausen,et al.  Quantile Regression Forests , 2006, J. Mach. Learn. Res..

[4]  Peter Buhlmann,et al.  BOOSTING ALGORITHMS: REGULARIZATION, PREDICTION AND MODEL FITTING , 2007, 0804.2752.

[5]  P. Bühlmann,et al.  Boosting With the L2 Loss , 2003 .

[6]  P. Bühlmann,et al.  Boosting with the L2-loss: regression and classification , 2001 .

[7]  Probal Chaudhuri,et al.  Nonparametric Estimates of Regression Quantiles and Their Local Bahadur Representation , 1991 .

[8]  Peter Buhlmann Boosting Methods: Why They Can Be Us eful for High-Dimensional Data , 2003 .

[9]  Thomas Lengauer,et al.  Permutation importance: a corrected feature importance measure , 2010, Bioinform..

[10]  Leo Breiman,et al.  Prediction Games and Arcing Algorithms , 1999, Neural Computation.

[11]  J. Friedman Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[12]  Bin Yu,et al.  Boosting with early stopping: Convergence and consistency , 2005, math/0508276.

[13]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[14]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[15]  Stephen Portnoy,et al.  Bivariate quantile smoothing splines , 1998 .

[16]  Brian S. Cade,et al.  Modeling Stream Fish Habitat Limitations from Wedge-Shaped Patterns of Variation in Standing Stock , 1996 .

[17]  Torsten Hothorn,et al.  Identifying Risk Factors for Severe Childhood Malnutrition by Boosting Additive Quantile Regression , 2011 .