Hidden Markov Model} Induction by Bayesian Model Merging

This paper describes a technique for learning both the number of states and the topology of Hidden Markov Models from examples. The induction process starts with the most specific model consistent with the training data and generalizes by successively merging states. Both the choice of states to merge and the stopping criterion are guided by the Bayesian posterior probability. We compare our algorithm with the Baum-Welch method of estimating fixed-size models, and find that it can induce minimal HMMs from data in cases where fixed estimation does not converge or requires redundant parameters to converge.

[1]  James Jay Horning,et al.  A study of grammatical inference , 1969 .

[2]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[3]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[4]  J. Rissanen.  A UNIVERSAL PRIOR FOR INTEGERS AND ESTIMATION BY MINIMUM DESCRIPTION LENGTH , 1983 .

[5]  Carl H. Smith,et al.  Inductive Inference: Theory and Methods , 1983, CSUR.

[6]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[7]  C. S. Wallace,et al.  Estimation and Inference by Compact Coding , 1987 .

[8]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[9]  M. V. Rossum,et al.  In Neural Computation , 2022 .

[10]  Steve Young,et al.  Applications of stochastic context-free grammars using the Inside-Outside algorithm , 1990 .

[11]  Stephen M. Omohundro,et al.  Best-First Model Merging for Dynamic Learning and Recognition , 1991, NIPS.

[12]  J. Feldman,et al.  Learning Automata from Ordered Examples , 1991 .

[13]  Wray L. Buntine,et al.  Learning classification trees , 1992 .

[14]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[15]  Pierre Baldi,et al.  Hidden Markov Models in Molecular Biology: New Algorithms and Applications , 1992, NIPS.

[16]  D. Haussler,et al.  Protein modeling using hidden Markov models: analysis of globins , 1993, [1993] Proceedings of the Twenty-sixth Hawaii International Conference on System Sciences.

[17]  Andreas Stolcke,et al.  Best-first Model Merging for Hidden Markov Model Induction , 1994, ArXiv.