Concentration inequalities for the exponential weighting method

The paper is concerned with recovering an unknown vector from noisy data with the help of a family of ordered smoothers [11]. The estimators within this family are aggregated based on the exponential weighting method and the performance of the aggregated estimate is measured by the excess risk controlling deviation of the square losses from the oracle risk. Based on natural statistical properties of ordered smoothers, we propose a novel method for obtaining concentration inequalities for the exponential weighting method.

[1]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[2]  C. L. Mallows.  Some comments on C_p , 1973 .

[3]  C. Reinsch,et al.  Oscillation matrices with spline smoothing , 1975 .

[4]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[5]  A. Kneip.  Ordered Linear Smoothers , 1994 .

[6]  C. Mallows.  More comments on C p , 1995 .

[7]  Arkadi Nemirovski,et al.  Topics in Non-Parametric Statistics , 2000 .

[8]  A. Juditsky,et al.  Functional aggregation for nonparametric regression , 2000 .

[9]  Yuhong Yang.  Combining Different Procedures for Adaptive Regression , 2000, Journal of Multivariate Analysis.

[10]  Olivier Catoni,et al.  Statistical learning theory and stochastic optimization , 2004 .

[11]  G. Lecu'e.  Simultaneous adaptation to the margin and to complexity in classification , 2005, math/0509696.

[12]  A. Tsybakov,et al.  Linear and convex aggregation of density estimators , 2006, math/0605292.

[13]  Andrew R. Barron,et al.  Information Theory and Mixing Least-Squares Regressions , 2006, IEEE Transactions on Information Theory.

[14]  Yu. Golubev.  On universal oracle inequalities related to high-dimensional linear models , 2010, 1011.2378.

[15]  A. Dalalyan,et al.  Sharp Oracle Inequalities for Aggregation of Affine Estimators , 2011, 1104.3969.

[16]  Karim Lounici,et al.  Pac-Bayesian Bounds for Sparse Regression Estimation with Exponential Weights , 2010, 1009.2707.

[17]  Karim Lounici,et al.  Variable Selection with Exponential Weights and $l_0$-Penalization , 2012, 1208.2635.

[18]  A. Tsybakov,et al.  Sparse Estimation by Exponential Weighting , 2011, 1108.5116.

[19]  G. K. Golubev.  Exponential weighting and oracle inequalities for projection estimates , 2012, Probl. Inf. Transm..

[20]  Yu. Golubev,et al.  Ordered smoothers with exponential weighting , 2012, 1211.4207.

[21]  C. Mallows.  Some Comments on Cp , 2000, Technometrics.