Concentration inequalities using the entropy method

We investigate a new methodology, worked out by Ledoux and Massart, to prove concentration-of-measure inequalities. The method is based on certain modified logarithmic Sobolev inequalities. We provide some very simple and general ready-to-use inequalities. One of these inequalities may be considered as an exponential version of the Efron--Stein inequality. The main purpose of this paper is to point out the simplicity and the generality of the approach. We show how the new method can recover many of Talagrand's revolutionary inequalities and provide new applications in a variety of problems including Rademacher averages, Rademacher chaos, the number of certain small subgraphs in a random graph, and the minimum of the empirical risk in some statistical estimation problems.

[1]  C. McDiarmid.  Concentration , 1862, The Dental register.

[2]  M. Sion.  On general minimax theorems , 1958 .

[3]  P. Gács,et al.  Bounds on conditional probabilities with applications in multi-user communication , 1976 .

[4]  J. Hoffmann-jorgensen.  Probability in Banach Space , 1977 .

[5]  B. Efron,et al.  The Jackknife Estimate of Variance , 1981 .

[6]  Béla Bollobás,et al.  Random Graphs , 1985 .

[7]  J. Steele.  An Efron-Stein inequality for nonsymmetric statistics , 1986 .

[8]  Katalin Marton,et al.  A simple proof of the blowing-up lemma , 1986, IEEE Trans. Inf. Theory.

[9]  J. Steele.  Probability theory and combinatorial optimization , 1987 .

[10]  Colin McDiarmid,et al.  Surveys in Combinatorics, 1989: On the method of bounded differences , 1989 .

[11]  Svante Janson,et al.  Poisson Approximation for Large Deviations , 1990, Random Struct. Algorithms.

[12]  M. Talagrand,et al.  Probability in Banach spaces , 1991 .

[13]  M. Talagrand.  Concentration of measure and isoperimetric inequalities in product spaces , 1994, math/9406212.

[14]  Jon A. Wellner,et al.  Weak Convergence and Empirical Processes: With Applications to Statistics , 1996 .

[15]  M. Talagrand.  New concentration inequalities in product spaces , 1996 .

[16]  K. Marton.  A measure concentration inequality for contracting markov chains , 1996 .

[17]  M. Ledoux,et al.  Isoperimetry and Gaussian analysis , 1996 .

[18]  M. Talagrand.  A new look at independence , 1996 .

[19]  A. Dembo.  Information inequalities and concentration of measure , 1997 .

[20]  M. Ledoux.  On Talagrand's deviation inequalities for product measures , 1997 .

[21]  E. Giné,et al.  Decoupling: From Dependence to Independence , 1998 .

[22]  M. Habib.  Probabilistic methods for algorithmic discrete mathematics , 1998 .

[23]  S. Bobkov,et al.  On Modified Logarithmic Sobolev Inequalities for Bernoulli and Poisson Measures , 1998 .

[24]  V. Vu.  On the Concentration of Multi-Variate Polynomials with Small Expectation , 2000 .

[25]  S. Boucheron,et al.  A sharp concentration inequality with applications , 1999, Random Struct. Algorithms.

[26]  P. Massart.  Some applications of concentration inequalities to statistics , 2000 .

[27]  P. Massart,et al.  About the constants in Talagrand's concentration inequalities for empirical processes , 2000 .

[28]  V. Vu.  On the concentration of multivariate polynomials with small expectation , 2000, Random Struct. Algorithms.

[29]  Van H. Vu,et al.  Concentration of Multivariate Polynomials and Its Applications , 2000, Comb..

[30]  M. Ledoux.  The concentration of measure phenomenon , 2001 .

[31]  E. Rio,et al.  Inégalités de concentration pour les processus empiriques de classes de parties , 2001 .

[32]  Van H. Vu,et al.  A Large Deviation Result on the Number of Small Subgraphs of a Random Graph , 2001, Combinatorics, Probability and Computing.

[33]  Svante Janson,et al.  The infamous upper tail , 2002, Random Struct. Algorithms.

[34]  V. Koltchinskii,et al.  Empirical margin distributions and bounding the generalization error of combined classifiers , 2002, math/0405343.

[35]  O. Bousquet.  A Bennett concentration inequality and its application to suprema of empirical processes , 2002 .

[36]  Peter L. Bartlett,et al.  Model Selection and Error Estimation , 2000, Machine Learning.

[37]  Svante Janson,et al.  The Deletion Method For Upper Tail Estimates , 2004, Comb..