Asymptotic Normality of the Maximum Pseudolikelihood Estimator for Fully Visible Boltzmann Machines

Boltzmann machines (BMs) are a class of binary neural networks for which there have been numerous proposed methods of estimation. Recently, it has been shown that in the fully visible case of the BM, the method of maximum pseudolikelihood estimation (MPLE) results in parameter estimates, which are consistent in the probabilistic sense. In this brief, we investigate the properties of MPLE for the fully visible BMs further, and prove that MPLE also yields an asymptotically normal parameter estimator. These results can be used to construct confidence intervals and to test statistical hypotheses. These constructions provide a closed-form alternative to the current methods that require Monte Carlo simulation or resampling. We support our theoretical results by showing that the estimator behaves as expected in simulation studies.

[1]  S. Mase,et al.  Consistency of the Maximum Pseudo-Likelihood Estimator of Continuous State Space Gibbsian Processes , 1995 .

[2]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[3]  Barry C. Arnold,et al.  PSEUDOLIKELIHOOD ESTIMATION : SOME EXAMPLES , 2016 .

[4]  Stuart Geman,et al.  Markov Random Field Image Models and Their Applications to Computer Vision , 2010 .

[5]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[6]  L. Brown,et al.  Interval Estimation for a Binomial Proportion , 2001 .

[7]  Aapo Hyvärinen,et al.  Some extensions of score matching , 2007, Comput. Stat. Data Anal..

[8]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[9]  Hilbert J. Kappen,et al.  Efficient Learning in Boltzmann Machines Using Linear Response Theory , 1998, Neural Computation.

[10]  Dennis D. Boos,et al.  Essential Statistical Inference , 2013 .

[11]  Dennis D. Boos,et al.  Essential Statistical Inference: Theory and Methods , 2013 .

[12]  C. Varin.  On composite marginal likelihoods , 2008 .

[13]  Stephen E. Fienberg,et al.  Testing Statistical Hypotheses , 2005 .

[14]  C. Varin,et al.  A note on composite likelihood inference and model selection , 2005 .

[15]  Lawrence D. Brown,et al.  INTERVAL ESTIMATION IN EXPONENTIAL FAMILIES , 2003 .

[16]  Aapo Hyvärinen,et al.  Consistency of Pseudolikelihood Estimation of Fully Visible Boltzmann Machines , 2006, Neural Computation.

[17]  Sherif Hashem.  Effects of Collinearity on Combining Neural Networks , 1996, Connect. Sci..

[18]  G. Marsaglia,et al.  Evaluating the Anderson-Darling Distribution , 2004 .

[19]  J. Besag.  Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[20]  D. Cox,et al.  A note on pseudolikelihood constructed from marginal densities , 2004 .

[21]  D. Pollard.  Asymptotics for Least Absolute Deviation Regression Estimators , 1991, Econometric Theory.

[22]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[23]  Shin Ishii,et al.  A Closed-Form Estimator of Fully Visible Boltzmann Machines , 2008, ICONIP.

[24]  G. Molenberghs,et al.  Models for Discrete Longitudinal Data , 2005 .

[25]  B. Desmarais.  Consistent Confidence Intervals for Maximum Pseudolikelihood Estimators , .

[26]  A. V. D. Vaart,et al.  Asymptotic Statistics: Frontmatter , 1998 .

[27]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[28]  William J. Byrne,et al.  Alternating minimization and Boltzmann machine learning , 1992, IEEE Trans. Neural Networks.

[29]  N. Reid,et al.  AN OVERVIEW OF COMPOSITE LIKELIHOOD METHODS , 2011 .

[30]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..