Concentration Inequalities for Samples without Replacement

This paper considers the concentration of values of functions of random variables sampled without replacement from a fixed finite set close to their expectations---a problem which is relevant to a variety of applications, including the transductive formulation of statistical learning theory. Apart from the review of known results, the paper studies two general approaches leading in many cases to sufficiently exact concentration inequalities. The first is based on the sub-Gaussian inequality of Bobkov [Ann. Probab., 32 (2004), pp. 2884--2907] for functions defined on a slice of the discrete cube. The second approach proposed by Hoeffding [J. Amer. Statist. Assoc., 58 (1963), pp. 13--30] reduces the problem to studying a sample of independent random variables.

[1]  P. Massart.  Rates of convergence in the central limit theorem for empirical processes , 1986 .

[2]  M. Talagrand.  New concentration inequalities in product spaces , 1996 .

[3]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[4]  V. Koltchinskii,et al.  Rademacher Processes and Bounding the Risk of Function Learning , 2004, math/0405338.

[5]  O. Bousquet.  Concentration Inequalities and Empirical Processes Theory Applied to the Analysis of Learning Algorithms , 2002 .

[6]  Sergey G. Bobkov.  Concentration of normalized sums and a central limit theorem for noncorrelated random variables , 2004 .

[7]  Ran El-Yaniv,et al.  Explicit Learning Curves for Transduction and Application to Clustering and Compression Algorithms , 2004, J. Artif. Intell. Res..

[8]  P. Bartlett,et al.  Local Rademacher complexities , 2005, math/0508275.

[9]  V. Koltchinskii.  Local Rademacher complexities and oracle inequalities in risk minimization , 2006, 0708.0083.

[10]  Ran El-Yaniv,et al.  Transductive Rademacher Complexity and Its Applications , 2007, COLT.

[11]  Vincent Nesme,et al.  Note on sampling without replacing from a finite collection of matrices , 2010, ArXiv.

[12]  Konstantin Vorontsov,et al.  Tight Combinatorial Generalization Bounds for Threshold Conjunction Rules , 2011, PReMI.

[13]  V. Koltchinskii,et al.  Oracle inequalities in empirical risk minimization and sparse recovery problems , 2011 .

[14]  Gábor Lugosi,et al.  Concentration Inequalities - A Nonasymptotic Theory of Independence , 2013, Concentration Inequalities.