Model-free multiclass conditional probability estimation via quantile regression

Conventional multiclass conditional probability estimation often requires distributional model assumptions such as Fisher's discriminate analysis and logistic regression. In this paper,a model-free estimation method is proposed to estimate multiclass conditional probability through a series of conditional quantile regression functions. Specifically, each quantile regression is constructed by optimizing a regularized formulation in a reproducing kernel Hilbert space, and then the conditional class probability can be well estimated by converting the quantile regression functions to the corresponding cumulative distribution functions. The primary advantage of the proposed estimation method is that it does not assume any distributional assumption and its computation cost does not increase with the number of classes. The theoretical and numerical studies demonstrate that the proposed estimation method is highly competitive against other alternatives, especially when the number of classes is relatively large.

[1]  Nicole A. Lazar,et al.  Quantile estimation for discrete data via empirical likelihood , 2010 .

[2]  Keming Yu,et al.  Quantile regression: applications and current research areas , 2003 .

[3]  Yufeng Liu,et al.  Simultaneous multiple non-crossing quantile regression estimation using kernel constraints , 2011, Journal of nonparametric statistics.

[4]  Xuming He,et al.  Bayesian empirical likelihood for quantile regression , 2012, 1207.5378.

[5]  Chong Gu.  Smoothing Spline Anova Models , 2002 .

[6]  W. Wong,et al.  Convergence Rate of Sieve Estimates , 1994 .

[7]  B. Efron.  The Estimation of Prediction Error , 2004 .

[8]  J. S. Silva,et al.  Quantiles for Counts , 2002 .

[9]  Saharon Rosset.  Bi-level path following for cross validated solution of kernel quantile regression , 2008, ICML '08.

[10]  Ji Zhu,et al.  Quantile Regression in Reproducing Kernel Hilbert Spaces , 2007 .

[11]  H. Bondell,et al.  Noncrossing quantile regression curve estimation. , 2010, Biometrika.

[12]  Xuming He.  Quantile Curves without Crossing , 1997 .

[13]  Hao Helen Zhang,et al.  Robust Model-Free Multiclass Probability Estimation , 2010, Journal of the American Statistical Association.

[14]  L. Breiman,et al.  Submodel selection and evaluation in regression. The X-random case , 1992 .

[15]  G. Wahba,et al.  Multicategory Support Vector Machines , Theory , and Application to the Classification of Microarray Data and Satellite Radiance Data , 2004 .

[16]  G. Wahba.  Spline models for observational data , 1990 .

[17]  Stephen Portnoy,et al.  Bivariate quantile smoothing splines , 1998 .

[18]  Xiaotong Shen,et al.  Optimal Model Assessment, Selection, and Combination , 2006 .

[19]  Takafumi Kanamori,et al.  Nonparametric Conditional Density Estimation Using Piecewise-Linear Solution Path of Kernel Quantile Regression , 2009, Neural Computation.

[20]  L. Breiman.  Heuristics of instability and stabilization in model selection , 1996 .

[21]  Yufeng Liu,et al.  Probability estimation for large-margin classifiers , 2008 .

[22]  Ding-Xuan Zhou,et al.  The covering number in learning theory , 2002, J. Complex..