Bi-level path following for cross validated solution of kernel quantile regression

Modeling of conditional quantiles requires specification of the quantile being estimated and can thus be viewed as a parameterized predictive modeling problem. Quantile loss is typically used, and it is indeed parameterized by a quantile parameter. In this paper we show how to follow the path of cross validated solutions to regularized kernel quantile regression. Even though the bi-level optimization problem we encounter for every quantile is non-convex, the manner in which the optimal cross-validated solution evolves with the parameter of the loss function allows tracking of this solution. We prove this property, construct the resulting algorithm, and demonstrate it on data. This algorithm allows us to efficiently solve the whole family of bi-level problems.

[1]  Andreas Christmann,et al.  How SVMs can estimate quantiles and the median , 2007, NIPS.

[2]  S. Rosset,et al.  Piecewise linear regularized solution paths , 2007, 0708.2197.

[3]  Alexander J. Smola,et al.  Nonparametric Quantile Estimation , 2006, J. Mach. Learn. Res..

[4]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[5]  David Mease,et al.  Boosted Classification Trees and Class Probability/Quantile Estimation , 2007, J. Mach. Learn. Res..

[6]  Nicolai Meinshausen,et al.  Quantile Regression Forests , 2006, J. Mach. Learn. Res..

[7]  R. Tibshirani,et al.  On the “degrees of freedom” of the lasso , 2007, 0712.0881.

[8]  Ming Yuan,et al.  GACV for quantile smoothing splines , 2006, Comput. Stat. Data Anal..

[9]  Gang Wang,et al.  Two-dimensional solution path for support vector regression , 2006, ICML.

[10]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[11]  Saharon Rosset,et al.  Bi-level path following for cross validated solution of kernel quantile regression , 2008, ICML '08.

[12]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[13]  Robert Tibshirani,et al.  The Entire Regularization Path for the Support Vector Machine , 2004, J. Mach. Learn. Res..

[14]  Micha Sharir,et al.  Davenport-Schinzel sequences and their geometric applications , 1995, Handbook of Computational Geometry.

[15]  G. Wahba,et al.  Some results on Tchebycheffian spline functions , 1971 .

[16]  Takafumi Kanamori,et al.  Nonparametric Conditional Density Estimation Using Piecewise-Linear Solution Path of Kernel Quantile Regression , 2009, Neural Computation.

[17]  R. Pace,et al.  Sparse spatial autoregressions , 1997 .

[18]  Jing Hu,et al.  Bilevel Model Selection for Support Vector Machines , 2007 .

[19]  Ji Zhu,et al.  Quantile Regression in Reproducing Kernel Hilbert Spaces , 2007 .

[20]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[21]  Eric R. Eide,et al.  The effect of school quality on student performance: A quantile regression approach , 1998 .

[22]  Moshe Buchinsky CHANGES IN THE U.S. WAGE STRUCTURE 1963-1987: APPLICATION OF QUANTILE REGRESSION , 1994 .

[23]  Bianca Zadrozny,et al.  High-quantile modeling for customer wallet estimation and other applications , 2007, KDD '07.

[24]  Stephen Portnoy,et al.  Bivariate quantile smoothing splines , 1998 .

[25]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[26]  Yufeng Liu,et al.  Probability estimation for large-margin classifiers , 2008 .

[27]  Andreas Christmann,et al.  Consistency of kernel-based quantile regression , 2008 .

[28]  Ji Zhu,et al.  Efficient Computation and Model Selection for the Support Vector Regression , 2007, Neural Computation.