Rates of contraction of posterior distributions based on Gaussian process priors

We derive rates of contraction of posterior distributions on nonparametric or semiparametric models based on Gaussian processes. The rate of contraction is shown to depend on the position of the true parameter relative to the reproducing kernel Hilbert space of the Gaussian process and the small ball probabilities of the Gaussian process. We determine these quantities for a range of examples of Gaussian priors and in several statistical settings. For instance, we consider the rate of contraction of the posterior distribution based on sampling from a smooth density model when the prior models the log density as a (fractionally integrated) Brownian motion. We also consider regression with Gaussian errors and smooth classification under a logistic or probit link function combined with various priors.

[1]  H. Kober.  ON FRACTIONAL INTEGRALS AND DERIVATIVES , 1940 .

[2]  B. Mandelbrot,et al.  Fractional Brownian Motions, Fractional Noises and Applications , 1968 .

[3]  G. Wahba,et al.  A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines , 1970 .

[4]  C. Borell.  The Brunn-Minkowski inequality in Gauss space , 1975 .

[5]  G. Wahba.  Improper Priors, Spline Smoothing and the Problem of Guarding Against Model Errors in Regression , 1978 .

[6]  J. Kuelbs.  Probability on Banach spaces , 1978 .

[7]  Tom Leonard.  Density Estimation, Stochastic Processes and Prior Information , 1978 .

[8]  P. Lenk.  The Logistic Normal Distribution for Bayesian, Nonparametric, Predictive Densities , 1988 .

[9]  A. V. D. Vaart,et al.  Statistical estimation in large parameter spaces , 1988 .

[10]  Peter J. Lenk,et al.  Towards a practicable Bayesian nonparametric density estimator , 1991 .

[11]  J. Kuelbs,et al.  Metric entropy and the small ball problem for Gaussian measures , 1993 .

[12]  S. Chib,et al.  Bayesian analysis of binary and polychotomous response data , 1993 .

[13]  J. Kuelbs,et al.  The Gaussian measure of shifted balls , 1994 .

[14]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[15]  Jon A. Wellner,et al.  Weak Convergence and Empirical Processes: With Applications to Statistics , 1996 .

[16]  L. Werner.  Existence of small ball constants for fractional Brownian motions , 1998 .

[17]  P. Lenk.  Bayesian inference for semiparametric regression using a Fourier representation , 1999 .

[18]  A. V. D. Vaart,et al.  Convergence rates of posterior distributions , 2000 .

[19]  Q. Shao,et al.  Gaussian processes: Inequalities, small ball probabilities and applications , 2001 .

[20]  A. V. D. Vaart,et al.  Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities , 2001 .

[21]  Thomas Kühn,et al.  Optimal series representation of fractional Brownian sheets , 2002 .

[22]  Kacha Dzhaparidze,et al.  A series expansion of fractional Brownian motion , 2002 .

[23]  S. Ghosal,et al.  On Bayesian Adaptation , 2003 .

[24]  Murad S. Taqqu,et al.  Rate Optimality of Wavelet Series Approximations of Fractional Brownian Motion , 2003 .

[25]  Henrik Hult,et al.  Approximating some volterra type stochastic integrals with applications to parameter estimation , 2003 .

[26]  M. Schervish,et al.  Posterior Consistency in Nonparametric Regression Problems under Gaussian Process Priors , 2004 .

[27]  Harry van Zanten,et al.  Krein's spectral theory and the Paley-Wiener expansion for fractional Brownian motion , 2005, math/0503656.

[28]  A Rate-Optimal Trigonometric Series Expansion of the Fractional Brownian Motion , 2005 .

[29]  S. Ghosal,et al.  Posterior consistency of Gaussian process prior for nonparametric binary regression , 2006, math/0702686.

[30]  Van Der Vaart,et al.  Convergence rates of posterior distributions for Brownian semimartingale models , 2006 .

[31]  A. V. D. Vaart,et al.  Misspecification in infinite-dimensional Bayesian statistics , 2006, math/0607023.

[32]  S. Ghosal,et al.  Nonparametric binary regression using a Gaussian process prior , 2007 .

[33]  A. V. D. Vaart,et al.  Convergence rates of posterior distributions for non-i.i.d. observations , 2007, 0708.0491.

[34]  J. Ghosh,et al.  Posterior consistency of logistic Gaussian process priors in density estimation , 2007 .

[35]  A. V. D. Vaart,et al.  Posterior convergence rates of Dirichlet mixtures at smooth densities , 2007, 0708.1885.

[36]  A. W. Vaart,et al.  Reproducing kernel Hilbert spaces of Gaussian priors , 2008, 0805.3252.

[37]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.