Ordered smoothers with exponential weighting

The main goal in this paper is to propose a new approach to deriving oracle inequalities related to the exponential weighting method. The paper focuses on recovering an unknown vector from noisy data with the help of the family of ordered smoothers. The estimators within this family are aggregated using the exponential weighting method and the aim is to control the risk of the aggregated estimate. Based on natural probabilistic properties of the unbiased risk estimate, we derive new oracle inequalities for mean square risk and show that the exponential weighting permits to improve Kneip's oracle inequality.

[1]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[2]  C. Reinsch,et al.  Oscillation matrices with spline smoothing , 1975 .

[3]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[4]  M. Nussbaum.  Spline Smoothing in Regression Models and Asymptotic Efficiency in $L_2$ , 1985 .

[5]  P. Speckman.  Spline Smoothing and Optimal Rates of Convergence in Nonparametric Regression Models , 1985 .

[6]  G. Wahba.  Spline models for observational data , 1990 .

[7]  B. Silverman,et al.  Nonparametric Regression and Generalized Linear Models: A roughness penalty approach , 1993 .

[8]  A. Kneip.  Ordered Linear Smoothers , 1994 .

[9]  Arkadi Nemirovski,et al.  Topics in Non-Parametric Statistics , 2000 .

[10]  A. Juditsky,et al.  Functional aggregation for nonparametric regression , 2000 .

[11]  Yuhong Yang.  Combining Different Procedures for Adaptive Regression , 2000, Journal of Multivariate Analysis.

[12]  Olivier Catoni,et al.  Statistical learning theory and stochastic optimization , 2004 .

[13]  Yuhong Yang.  Aggregating regression procedures to improve performance , 2004 .

[14]  G. Lecu'e.  Simultaneous adaptation to the margin and to complexity in classification , 2005, math/0509696.

[15]  A. Tsybakov,et al.  Linear and convex aggregation of density estimators , 2006, math/0605292.

[16]  Andrew R. Barron,et al.  Information Theory and Mixing Least-Squares Regressions , 2006, IEEE Transactions on Information Theory.

[17]  P. Massart,et al.  Minimal Penalties for Gaussian Model Selection , 2007 .

[18]  Yu. Golubev.  On universal oracle inequalities related to high-dimensional linear models , 2010, 1011.2378.

[19]  A. Dalalyan,et al.  Sharp Oracle Inequalities for Aggregation of Affine Estimators , 2011, 1104.3969.

[20]  Karim Lounici,et al.  Pac-Bayesian Bounds for Sparse Regression Estimation with Exponential Weights , 2010, 1009.2707.

[21]  Yu. Golubev.  Exponential weighting and oracle inequalities for projection methods , 2012, 1206.4285.

[22]  A. Tsybakov,et al.  Sparse Estimation by Exponential Weighting , 2011, 1108.5116.

[23]  G. K. Golubev.  Exponential weighting and oracle inequalities for projection estimates , 2012, Probl. Inf. Transm..