Third-order coding rate for universal compression of Markov sources

We consider the universal source coding problem for first-order stationary, irreducible and aperiodic Markov sources for short blocklengths. Achievability is derived based on the previously introduced algorithm for universal compression of memoryless sources in the finite blocklengths, the Type Size Code, which encodes strings based on type class size. We derive the third-order asymptotic coding rate of the Type Size code for this model class. We also present a converse on the third-order coding rate for the general class of fixed-to-variable codes and show the optimality of Type Size codes for such Markov sources.

[1]  Shigeichi Hirasawa,et al.  A Note on the overflow probability of lossless codes , 2006 .

[2]  Toshiyasu Matsushima,et al.  Evaluation of the minimum overflow threshold of bayes codes for a Markov source , 2014, 2014 International Symposium on Information Theory and its Applications.

[3]  Giuseppe Longo,et al.  The error exponent for the noiseless encoding of finite ergodic Markov sources , 1981, IEEE Trans. Inf. Theory.

[4]  Oliver Kosut,et al.  Universal fixed-to-variable source coding in the finite blocklength regime , 2013, 2013 IEEE International Symposium on Information Theory.

[5]  Neri Merhav,et al.  Optimal sequential probability assignment for individual sequences , 1994, IEEE Trans. Inf. Theory.

[6]  P. Whittle,et al.  Some Distribution and Moment Formulae for the Markov Chain , 1955 .

[7]  Oliver Kosut,et al.  Asymptotics and Non-Asymptotics for Universal Fixed-to-Variable Source Coding , 2014, IEEE Transactions on Information Theory.

[8]  T. W. Anderson.  Second-Order Moments of a Stationary Markov Chain and Some Applications , 1989 .

[9]  Claudia Biermann,et al.  Mathematical Methods Of Statistics , 2016 .

[10]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[11]  Faramarz Fekri,et al.  Fundamental limits of universal lossless one-to-one compression of parametric sources , 2014, 2014 IEEE Information Theory Workshop (ITW 2014).

[12]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[13]  Sergio Verdú,et al.  Optimal Lossless Data Compression: Non-Asymptotics and Asymptotics , 2014, IEEE Transactions on Information Theory.

[14]  R. Lapinskas.  On the rate of convergence in a multidimensional central limit theorem for inhomogeneous Markov chains , 1974 .

[15]  de Ng Dick Bruijn,et al.  Circuits and Trees in Oriented Linear Graphs , 1951 .

[16]  L. B. Boza.  Asymptotically Optimal Tests for Finite Markov Chains , 1971 .

[17]  Oliver Kosut,et al.  New results on third-order coding rate for universal fixed-to-variable source coding , 2014, 2014 IEEE International Symposium on Information Theory.

[18]  J. Nicholas Laneman,et al.  A Finite-Blocklength Perspective on Gaussian Multi-Access Channels , 2013, ArXiv.