Approximate Confidence and Prediction Intervals for Least Squares Support Vector Regression

Bias-corrected approximate 100(1-α)% pointwise and simultaneous confidence and prediction intervals for least squares support vector machines are proposed. A simple way of determining the bias without estimating higher order derivatives is formulated. A variance estimator is developed that works well in the homoscedastic and heteroscedastic case. In order to produce simultaneous confidence intervals, a simple Šidák correction and a more involved correction (based on upcrossing theory) are used. The obtained confidence intervals are compared to a state-of-the-art bootstrap-based method. Simulations show that the proposed method obtains similar intervals compared to the bootstrap at a lower computational cost.

[1]  Changbao Wu,et al.  Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis , 1986 .

[2]  Léon Personnaz,et al.  Construction of confidence intervals for neural networks based on least squares estimation , 2000, Neural Networks.

[3]  S. Rice.  The Distribution of the Maxima of a Random Curve , 1939 .

[4]  Michael I. Jordan,et al.  Regression with input-dependent noise: A Gaussian process treatment , 1998 .

[5]  Paul L. Speckman,et al.  Confidence bands in nonparametric regression , 1993 .

[6]  P. Hall.  On Bootstrap Confidence Intervals in Nonparametric Regression , 1992 .

[7]  E. Mammen.  Bootstrap and Wild Bootstrap for High Dimensional Linear Models , 1993 .

[8]  M. Wand,et al.  Semiparametric Regression: Parametric Regression , 2003 .

[9]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[10]  A. Shiryaev,et al.  Probability (2nd ed.) , 1995, Technometrics.

[11]  James Stephen Marron,et al.  Regression smoothing parameters that are not far from their optimum , 1992 .

[12]  Wolfram Burgard,et al.  Most likely heteroscedastic Gaussian process regression , 2007, ICML '07.

[13]  Marcin Witczak.  Modelling and Estimation Strategies for Fault Diagnosis of Non-Linear Systems: From Analytical to Soft Computing Approaches , 2007 .

[14]  Peter Tino,et al.  IEEE Transactions on Neural Networks , 2009 .

[15]  C. Loader,et al.  Simultaneous Confidence Bands for Linear Regression and Smoothing , 1994 .

[16]  Robert Tibshirani,et al.  Correction: Discussion of "Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis" by C. F. J. Wu , 1988 .

[17]  D. M. Titterington,et al.  On confidence bands in nonparametric density estimation and regression , 1988 .

[18]  N. Weber,et al.  Discussion: Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis , 1986 .

[19]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .

[20]  Jianqing Fan,et al.  Local polynomial modelling and its applications , 1994 .

[21]  Jerome Sacks,et al.  Confidence Bands for Regression Functions , 1985 .

[22]  Gavin C. Cawley,et al.  Estimating Predictive Variances with Kernel Ridge Regression , 2005, MLCW.

[23]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[24]  W. Haerdle.  Resampling for inference from curves , 1989 .

[25]  Zhe Sun,et al.  Cutting Plane Method for Continuously Constrained Kernel-Based Regression , 2010, IEEE Transactions on Neural Networks.

[26]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[27]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[28]  George Chryssolouris,et al.  Confidence interval prediction for neural network models , 1996, IEEE Trans. Neural Networks.

[29]  Jianqing Fan,et al.  Data‐Driven Bandwidth Selection in Local Polynomial Fitting: Variable Bandwidth and Spatial Adaptation , 1995 .

[30]  Gordon J Johnston,et al.  Probabilities of maximal deviations for nonparametric regression function estimates , 1982 .

[31]  Wolfgang Karl Härdle,et al.  Local polynomial estimators of the volatility function in nonparametric autoregression , 1997 .

[32]  Regina Y. Liu.  Bootstrap Procedures under some Non-I.I.D. Models , 1988 .

[33]  P. Bickel,et al.  On Some Global Measures of the Deviations of Density Function Estimates , 1973 .

[34]  Rudolf Beran.  Discussion: Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis , 1986 .

[35]  Wolfgang Karl Härdle,et al.  Asymptotic maximal deviation of M-smoothers , 1989 .

[36]  Guohua Pan,et al.  Local Regression and Likelihood , 1999, Technometrics.

[37]  James Davidson,et al.  Implementing the wild bootstrap using a two-point distribution☆ , 2007 .

[38]  M. Wand.  Local Regression and Likelihood , 2001 .

[39]  Christopher M. Bishop,et al.  Regression with Input-Dependent Noise: A Bayesian Treatment , 1996, NIPS.

[40]  Jianbo Yang,et al.  An Improved Algorithm for the Solution of the Regularization Path of Support Vector Machine , 2010, IEEE Transactions on Neural Networks.

[41]  Alexandre B. Tsybakov,et al.  Introduction to Nonparametric Estimation , 2008, Springer series in statistics.

[42]  Jianqing Fan,et al.  Efficient Estimation of Conditional Variance Functions in Stochastic Regression , 1998 .

[43]  J. Rice.  Mathematical Statistics and Data Analysis , 1988 .

[44]  Alan F. Murray,et al.  Confidence estimation methods for neural networks : a practical comparison , 2001, ESANN.

[45]  Gavin C. Cawley,et al.  Heteroscedastic kernel ridge regression , 2004, Neurocomputing.

[46]  Richard A. Davis,et al.  On Some Global Measures of the Deviations of Density Function Estimates , 2011 .

[47]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[49]  Aarnout Brombacher,et al.  Probability... , 2009, Qual. Reliab. Eng. Int..