A Variational Method for Estimating the Parameters of MRF from Complete or Incomplete Data

We introduce a new method (to be referred to as the variational method, VM) for estimating the parameters of Gibbs distributions with random variables ("spins") taking values in a Euclidean space Rn, n > 1, from complete or degraded data. The method applies also to the case of iid random variables governed by exponential families, and appears to be new even in this case. For complete data, the VM is computationally more efficient than, and as reliable as, the maximum pseudo-likelihood method. For incomplete data, the VM leads to an estimation procedure reminiscent of, but simpler than, the EM algorithm. In the former case, we show that under natural assumptions a certain form of the variational estimators is strongly consistent and asymptotically normal. We also present numerical experiments that demonstrate the computational efficiency and accuracy of the variational estimators.

[1]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[2]  David Ruelle,et al.  Superstable interactions in classical statistical mechanics , 1970 .

[3]  J. Besag.  Statistical Analysis of Non-Lattice Data , 1975 .

[4]  D. Ruelle.  Probability estimates for continuous spin systems , 1976 .

[5]  J. Lebowitz,et al.  Statistical mechanics of systems of unbounded spins , 1976 .

[6]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[7]  A strong variational principle for continuous spin systems , 1980 .

[8]  H. Künsch.  Almost sure entropy and the variational principle for random fields with unbounded state space , 1981 .

[9]  Erratum: ``Statistical mechanics of systems of unbounded spins'' , 1981 .

[10]  Geoffrey E. Hinton,et al.  OPTIMAL PERCEPTUAL INFERENCE , 1983 .

[11]  L. Cobb,et al.  Estimation and Moment Recursion Relations for Multimodal Distributions of the Exponential Family , 1983 .

[12]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  B. Gidas.  Nonstationary Markov chains and convergence of the annealing algorithm , 1985 .

[14]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[15]  B. Gidas.  Global optimization via the Langevin equation , 1985, 1985 24th IEEE Conference on Decision and Control.

[16]  S. Geman,et al.  Diffusions for global optimizations , 1986 .

[17]  Haluk Derin,et al.  Modeling and Segmentation of Noisy and Textured Images Using Gibbs Random Fields , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  C. Ray Smith,et al.  Maximum-entropy and Bayesian methods in science and engineering , 1988 .

[19]  B. Gidas.  Consistency of Maximum Likelihood and Pseudo-Likelihood Estimators for Gibbs Distributions , 1988 .

[20]  A. Lippman.  A Maximum Entropy Method for Expert System Construction , 1988 .

[21]  Raj Reddy,et al.  Large-vocabulary speaker-independent continuous speech recognition: the sphinx system , 1988 .

[22]  B. Ripley.  Statistical inference for spatial processes , 1990 .

[23]  Basilis Gidas,et al.  Parameter Estimation for Gibbs Distributions from Partially Observed Data , 1992 .

[24]  Basilis Gidas,et al.  A Renormalization Group Approach to Image Processing Problems , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[26]  J. D. Wilson,et al.  A smoothed EM approach to indirect estimation problems, with particular reference to stereology and emission tomography , 1990 .

[27]  K. Lange.  Convergence of EM image reconstruction algorithms with Gibbs smoothing. , 1990, IEEE transactions on medical imaging.

[28]  Basilis Gidas,et al.  Asymptotics of maximum likelihood estimators for the Curie-Weiss model , 1991 .