Taylor series expansions for the entropy rate of Hidden Markov Processes

Finding the entropy rate of Hidden Markov Processes is an active research topic, of both theoretical and practical importance. A recently used approach is studying the asymptotic behavior of the entropy rate in various regimes. In this paper we generalize and prove a previous conjecture relating the entropy rate to entropies of finite systems. We use the proof to establish series expansions for the entropy rate in two different regimes. We also study the radius of convergence of the two series expansions.

[1]  Eytan Domany,et al.  Asymptotics of the entropy rate for a hidden Markov process , 2005, Data Compression Conference.

[2]  Tsachy Weissman,et al.  On the optimality of symbol-by-symbol filtering and denoising , 2004, IEEE Transactions on Information Theory.

[3]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[4]  Brian H. Marcus,et al.  Analyticity of Entropy Rate of Hidden Markov Chains , 2005, IEEE Transactions on Information Theory.

[5]  Tsachy Weissman,et al.  New bounds on the entropy rate of hidden Markov processes , 2004, Information Theory Workshop.

[6]  Or Zuk.  The Relative Entropy Rate For Two Hidden Markov Processes , 2006 .

[7]  József Lörinczi,et al.  Transformations of Gibbs measures , 1998 .

[8]  O. Zuk.  The Relative Entropy Rate For Two Hidden Markov Processes , 2007 .

[9]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[10]  Philippe Jacquet,et al.  On the entropy of a hidden Markov process , 2004, Data Compression Conference, 2004. Proceedings. DCC 2004.

[11]  Neri Merhav,et al.  Hidden Markov processes , 2002, IEEE Trans. Inf. Theory.

[12]  R. Dobrushin,et al.  Completely analytical interactions: Constructive description , 1987 .

[13]  Eytan Domany,et al.  The Entropy of a Binary Hidden Markov Process , 2005, ArXiv.

[14]  Brian H. Marcus,et al.  Analyticity of entropy rate in families of hidden markov chains , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[15]  Tsachy Weissman,et al.  Asymptotic filtering and entropy rate of a hidden Markov process in the rare transitions regime , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[16]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .