A Bernstein-Von Mises Theorem for discrete probability distributions

We investigate the asymptotic normality of the posterior distri- bution in the discrete setting, when model dimension increases with sample size. We consider a probability mass function �0 on \{0} and a sequence of truncation levels (kn)n satisfying k 3 nninfikn �0(i). Let ˆ � denote the maximum likelihood estimate of (�0(i))ikn and letn(�0) denote the kn- dimensional vector which i-th coordinate is defined by p n(ˆ �n(i) �0(i)) for 1 � ikn. We check that under mild conditions on �0 and on the sequence of prior probabilities on the kn-dimensional simplices, after centering and rescaling, the variation distance between the posterior distribution recen- tered around ˆ �n and rescaled by p n and the kn-dimensional Gaussian dis- tribution N(�n(�0),I 1 (�0)) converges in probability to 0. This theorem can be used to prove the asymptotic normality of Bayesian estimators of Shannon and Renyi entropies. The proofs are based on concentration inequalities for centered and non- centered Chi-square (Pearson) statistics. The latter allow to establish pos- terior concentration rates with respect to Fisher distance rather than with respect to the Hellinger distance as it is commonplace in non-parametric Bayesian statistics.