Consistency and robustness of kernel based regression

We investigate properties of kernel based regression (KBR) methods which are inspired by the convex risk minimization method of support vector machines. We first describe the relation between the used loss function of the KBR method and the tail of the response variable Y . We then establish a consistency result for KBR and give assumptions for the existence of the influence function. In particular, our results allow to choose the loss function and the kernel to obtain computational tractable and consistent KBR methods having bounded influence functions. Furthermore, bounds for the sensitivity curve which is a finite sample version of the influence function are developed, and some numerical experiments are discussed.

[1]  F. Browder.  Nonlinear functional analysis , 1970 .

[2]  J. Diestel,et al.  On vector measures , 1974 .

[3]  F. Hampel.  The Influence Curve and Its Role in Robust Estimation , 1974 .

[4]  J. Hoffmann-jorgensen.  Sums of independent Banach space valued random variables , 1974 .

[5]  J. Kuelbs.  Probability on Banach spaces , 1978 .

[6]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[7]  I. Ekeland,et al.  Infinite-Dimensional Optimization And Convexity , 1983 .

[8]  Peter J. Rousseeuw,et al.  ROBUST REGRESSION BY MEANS OF S-ESTIMATORS , 1984 .

[9]  P. Rousseeuw.  Least Median of Squares Regression , 1984 .

[10]  V. Yohai.  HIGH BREAKDOWN-POINT AND HIGH EFFICIENCY ROBUST ESTIMATES FOR REGRESSION , 1987 .

[11]  C. Jennison,et al.  Robust Statistics: The Approach Based on Influence Functions , 1987 .

[12]  D. Ruppert.  Robust Statistics: The Approach Based on Influence Functions , 1987 .

[13]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[14]  R. Phelps.  Convex Functions, Monotone Operators and Differentiability , 1989 .

[15]  Ruben H. Zamar,et al.  A Procedure for Robust Estimation and Inference in Linear Regression , 1991 .

[16]  Heinz Bauer,et al.  Maß- und Integrationstheorie , 1992 .

[17]  P. L. Davies.  Aspects of Robust Linear Regression , 1993 .

[18]  吉野 崇,et al.  Introduction to operator theory , 1993 .

[19]  J. Diestel,et al.  Absolutely Summing Operators , 1995 .

[20]  David E. Tyler,et al.  Constrained M-Estimation for Regression , 1996 .

[21]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[22]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[23]  P. Rousseeuw,et al.  Halfspace Depth and Regression Depth Characterize the Empirical Distribution , 1999 .

[24]  Regina Y. Liu,et al.  Regression depth. Commentaries. Rejoinder , 1999 .

[25]  Tong Zhang,et al.  Convergence of Large Margin Separable Linear Classification , 2000, NIPS.

[26]  Ingo Steinwart,et al.  On the Influence of the Kernel on the Consistency of Support Vector Machines , 2002, J. Mach. Learn. Res..

[27]  T. Poggio,et al.  Statistical Learning: Stability is Sufficient for Generalization and Necessary and Sufficient for Consistency of Empirical Risk Minimization , 2002 .

[28]  Dudley,et al.  Real Analysis and Probability: Measurability: Borel Isomorphism and Analytic Sets , 2002 .

[29]  Adam Krzyzak,et al.  A Distribution-Free Theory of Nonparametric Regression , 2002, Springer series in statistics.

[30]  Ingo Steinwart,et al.  Sparseness of Support Vector Machines , 2003, J. Mach. Learn. Res..

[31]  T. Poggio,et al.  General conditions for predictivity in learning theory , 2004, Nature.

[32]  Andreas Christmann,et al.  An approach to model complex high–dimensional insurance data , 2004 .

[33]  Lorenzo Rosasco,et al.  Some Properties of Regularized Kernel Methods , 2004, J. Mach. Learn. Res..

[34]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.

[35]  Andreas Christmann,et al.  On Robustness Properties of Convex Risk Minimization Methods for Pattern Recognition , 2004, J. Mach. Learn. Res..

[36]  Ingo Steinwart,et al.  Consistency of support vector machines and other regularized kernel classifiers , 2005, IEEE Transactions on Information Theory.

[37]  R. Shah,et al.  Least Squares Support Vector Machines , 2022 .