A nonasymptotic law of iterated logarithm for robust online estimators

In this paper, we provide tight deviation bounds for M-estimators, which are valid with a prescribed probability for every sample size. M-estimators are ubiquitous in machine learning and statistical learning theory. They are used both for defining prediction strategies and for evaluating their precision. Our deviation bounds can be seen as a non-asymptotic version of the law of iterated logarithm. They are established under general assumptions such as Lipschitz continuity of the loss function and (local) curvature of the population risk. These conditions are satisfied for most examples used in machine learning, including those that are known to be robust to outliers and to heavy tailed distributions. To further highlight the scope of applicability of the obtained results, a new algorithm, with provably optimal theoretical guarantees, for the best arm identification in a stochastic multi-arm bandit setting is presented. Numerical experiments illustrating the validity of the algorithm are reported.

[1]  Martin J. Wainwright,et al.  A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers , 2009, NIPS.

[2]  A. Khintchine Über einen Satz der Wahrscheinlichkeitsrechnung , 1924 .

[3]  Ohad Shamir,et al.  Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization , 2011, ICML.

[4]  P. Rigollet,et al.  Optimal learning with Q-aggregation , 2013, 1301.6080.

[5]  Jieping Ye,et al.  Scaling SVM and Least Absolute Deviations via Exact Data Reduction , 2013, ICML.

[6]  Alan Malek,et al.  Best Arm Identification for Contaminated Bandits , 2018, J. Mach. Learn. Res..

[7]  Gábor Lugosi,et al.  Concentration Inequalities - A Nonasymptotic Theory of Independence , 2013, Concentration Inequalities.

[8]  M. A. Arcones,et al.  Some strong limit theorems for M-estimators , 1994 .

[9]  Nathan Srebro,et al.  Fast Rates for Regularized Objectives , 2008, NIPS.

[10]  Matthew Malloy,et al.  lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits , 2013, COLT.

[11]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[12]  G. Wang,et al.  LAW OF THE ITERATED LOGARITHM AND INVARIANCE PRINCIPLE FOR M-ESTIMATORS , 1995 .

[13]  Upper Bounds on Asymptotic Variances of $M$-Estimators of Location , 1977 .

[14]  A. V. D. Vaart,et al.  Asymptotic Statistics: Frontmatter , 1998 .

[15]  Daniel J. Hsu,et al.  Loss Minimization and Parameter Estimation with Heavy Tails , 2013, J. Mach. Learn. Res..

[16]  R. Maronna Robust $M$-Estimators of Multivariate Location and Scatter , 1976 .

[17]  V. Koltchinskii,et al.  Oracle inequalities in empirical risk minimization and sparse recovery problems , 2011 .

[18]  S. Portnoy Asymptotic behavior of M-estimators of p regression parameters when p , 1985 .

[19]  Po-Ling Loh,et al.  Statistical consistency and asymptotic normality for high-dimensional robust M-estimators , 2015, ArXiv.

[20]  A. Kolmogoroff Über das Gesetz des iterierten Logarithmus , 1929 .

[21]  R. Munos,et al.  Best Arm Identification in Multi-Armed Bandits , 2010, COLT.

[22]  R. H. Farrell Asymptotic Behavior of Expected Sample Size in Certain One Sided Tests , 1964 .

[23]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.