Regularization, sparse recovery, and median-of-means tournaments

A regularized risk minimization procedure for regression function estimation is introduced that achieves near optimal accuracy and confidence under general conditions, including heavy-tailed predictor and response variables. The procedure is based on median-of-means tournaments, introduced by the authors in [8]. It is shown that the new procedure outperforms standard regularized empirical risk minimization procedures such as lasso or slope in heavy-tailed problems.

[1]  J. Hoffmann-jorgensen.  Probability in Banach Space , 1977 .

[2]  Jon A. Wellner,et al.  Weak Convergence and Empirical Processes: With Applications to Statistics , 1996 .

[3]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[4]  A. Goldenshluger.  On Spatial Adaptive Estimation of Nonparametric Regression , 2004 .

[5]  V. Koltchinskii,et al.  Oracle inequalities in empirical risk minimization and sparse recovery problems , 2011 .

[6]  Jean-Yves Audibert,et al.  Robust linear least squares regression , 2010, 1010.0074.

[7]  M. Lerasle,et al.  ROBUST EMPIRICAL MEAN ESTIMATORS , 2011, 1112.3914.

[8]  S. Mendelson,et al.  Learning subgaussian classes : Upper and minimax bounds , 2013, 1305.4825.

[9]  Daniel J. Hsu,et al.  Approximate loss minimization with heavy tails , 2013, ArXiv.

[10]  S. Mendelson,et al.  Compressed sensing under weak moment assumptions , 2014, 1401.2188.

[11]  Shahar Mendelson,et al.  Learning without Concentration , 2014, COLT.

[12]  S. Mendelson.  Upper bounds on product and multiplier empirical processes , 2014, 1410.8003.

[13]  Emmanuel J. Candès,et al.  SLOPE is Adaptive to Unknown Sparsity and Asymptotically Minimax , 2015, ArXiv.

[14]  Weijie J. Su,et al.  SLOPE-ADAPTIVE VARIABLE SELECTION VIA CONVEX OPTIMIZATION. , 2014, The annals of applied statistics.

[15]  S. Mendelson.  On aggregation for heavy-tailed classes , 2015, Probability Theory and Related Fields.

[16]  G. Lugosi,et al.  Empirical risk minimization for heavy-tailed losses , 2014, 1406.2462.

[17]  Stanislav Minsker.  Geometric median and robust estimation in Banach spaces , 2013, 1308.1334.

[18]  Shahar Mendelson,et al.  `local' vs. `global' parameters -- breaking the gaussian complexity barrier , 2015, 1504.02191.

[19]  S. Mendelson,et al.  Regularization and the small-ball method I: sparse recovery , 2016, 1601.05584.

[20]  G. Lugosi,et al.  Risk minimization by median-of-means tournaments , 2016, Journal of the European Mathematical Society.

[21]  S. Mendelson.  An optimal unrestricted learning procedure , 2017, 1707.05342.

[22]  Lecu'e Guillaume,et al.  Learning from MOM's principles , 2017 .

[23]  S. Mendelson.  On Multiplier Processes Under Weak Moment Assumptions , 2016, 1601.06523.

[24]  A. Tsybakov,et al.  Slope meets Lasso: Improved oracle bounds and optimality , 2016, The Annals of Statistics.

[25]  Soumendu Sundar Mukherjee,et al.  Weak convergence and empirical processes , 2019 .

[26]  Lecu'e Guillaume,et al.  Learning from MOM’s principles: Le Cam’s approach , 2017, Stochastic Processes and their Applications.

[27]  G. Lugosi,et al.  Sub-Gaussian estimators of the mean of a random vector , 2017, The Annals of Statistics.