Generalization bounds for nonparametric regression with $\beta-$mixing samples

In this paper we present a series of results that permit to extend in a direct manner uniform deviation inequalities of the empirical process from the independent to the dependent case characterizing the additional error in terms of β−mixing coefficients associated to the training sample. We then apply these results to some previously obtained inequalities for independent samples associated to the deviation of the least-squared error in nonparametric regression to derive corresponding generalization bounds for regression schemes in which the training sample may not be independent. These results provide a framework to analyze the error associated to regression schemes whose training sample comes from a large class of β−mixing sequences, including geometrically ergodic Markov samples, using only the independent case. More generally, they permit a meaningful extension of the Vapnik-Chervonenkis and similar theories for independent training samples to this class of β−mixing samples. Corresponding author. Email: juandavid.barreracano@epfl.ch. CMAP, École Polytechnique, Route de Saclay, 91128 Palaiseau cedex, France and SDS, École Polytechnique Fédérale de Lausanne. EPFL SB MATH MA C2 647 (Bâtiment MA) Station 8 CH-1015 Lausanne, Switzerland. Supported in 2019 by the Chaire Marchés en Mutation, Fédération Française Bancaire and by the Institut Louis Bachelier. Email: emmanuel.gobet@polytechnique.edu. CMAP, Ecole Polytechnique, Route de Saclay, 91128 Palaiseau cedex, France. The authors research is part of the Chair Financial Risks of the Risk Foundation and the Finance for Energy Market Research Centre. This research also benefited from the support of the Chair Stress Test, RISK Management and Financial Steering, led by the French École Polytechnique and its Foundation and sponsored by BNP Paribas.

[1]  É. Moulines,et al.  Polynomial ergodicity of Markov transition kernels , 2003 .

[2]  Jon A. Wellner,et al.  Weak Convergence and Empirical Processes: With Applications to Statistics , 1996 .

[3]  R. Douc,et al.  Practical drift conditions for subgeometric rates of convergence , 2004, math/0407122.

[4]  David Barrera,et al.  Quantitative bounds for concentration-of-measure inequalities and empirical regression: The independent case , 2019, J. Complex..

[5]  M. Talagrand,et al.  Probability in Banach Spaces: Isoperimetry and Processes , 1991 .

[6]  Bin Yu.  RATES OF CONVERGENCE FOR EMPIRICAL PROCESSES OF STATIONARY MIXING SEQUENCES , 1994 .

[7]  S. Shelah.  A combinatorial problem; stability and order for models and theories in infinitary languages. , 1972 .

[8]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[9]  J. Dedecker,et al.  Subgaussian concentration inequalities for geometrically ergodic Markov chains , 2014, 1412.1794.

[10]  A. Sapozhnikov.  Subgeometric rates of convergence of f-ergodic Markov chains , 2006 .

[11]  R. C. Bradley.  Basic properties of strong mixing conditions. A survey and some open questions , 2005, math/0511078.

[12]  Ambuj Tewari,et al.  Sequential complexities and uniform martingale laws of large numbers , 2015 .

[13]  R. Syski,et al.  Random Walks With Stationary Increments and Renewal Theory , 1982 .

[14]  Gersende Fort,et al.  Convergence of the Monte Carlo expectation maximization for curved exponential families , 2003 .

[15]  A. Dembo,et al.  A note on uniform laws of averages for dependent processes , 1993 .

[16]  D. Pollard.  Empirical Processes: Theory and Applications , 1990 .

[17]  Mehryar Mohri,et al.  Rademacher Complexity Bounds for Non-I.I.D. Processes , 2008, NIPS.

[18]  G. Viennet.  Inequalities for absolutely regular sequences: application to density estimation , 1997 .

[19]  P. Doukhan,et al.  Weak Dependence: With Examples and Applications , 2007 .

[20]  Mehryar Mohri,et al.  Generalization bounds for non-stationary mixing processes , 2016, Machine Learning.

[21]  Adam Krzyzak,et al.  A Distribution-Free Theory of Nonparametric Regression , 2002, Springer series in statistics.

[22]  R. Adamczak.  A tail inequality for suprema of unbounded empirical processes with applications to Markov chains , 2007, 0709.3110.

[23]  E. Rio,et al.  Bernstein inequality and moderate deviations under strong mixing conditions , 2012, 1202.4777.

[24]  Eduardo D. Sontag,et al.  Feedforward Nets for Interpolation and Classification , 1992, J. Comput. Syst. Sci..

[25]  R. Douc,et al.  Subgeometric rates of convergence of f-ergodic strong Markov processes , 2006, math/0605791.

[26]  J. Yukich.  Rates of convergence for classes of functions: the non-i.i.d. case , 1986 .

[27]  P. Doukhan.  Mixing: Properties and Examples , 1994 .

[28]  G. Roberts,et al.  Polynomial convergence rates of Markov chains. , 2002 .

[29]  Gersende Fort,et al.  MCMC design-based non-parametric regression for rare event. Application to nested risk computations , 2017, Monte Carlo Methods Appl..

[30]  Norbert Sauer,et al.  On the Density of Families of Sets , 1972, J. Comb. Theory A.

[31]  Majid Mojirsheibani,et al.  A Note on Nonparametric Regression with β-Mixing Sequences , 2010 .