A deterministic annealing approach to discriminative hidden Markov model design

We present the problem of designing a classifier system based on hidden Markov models (HMMs) from a labeled training set with the objective of minimizing the rate of misclassification. To design the globally optimal recognizer, all the HMMs must be jointly optimized to minimize the number of mis-classified training patterns. This is a difficult design problem which we attack using the technique of deterministic annealing (DA). In the DA approach, we introduce randomness in the classification rule and minimize the expected mis-classification rate of the random classifier while controlling the level of randomness in its decision via a constraint on the Shannon entropy. The effective cost function is smooth and converges to the mis-classification cost at the limit of zero entropy (non-random classification rule). The DA approach can be implemented via an efficient forward-backward algorithm for recomputing the model parameters. This algorithm significantly outperforms the standard maximum likelihood algorithm for a moderate increase in design complexity.