Universal coding with point type classes

With a customized characterization of types, every universal one-to-one coding algorithm can be described as follows: assign sequences to binary strings based on their type class sizes from smallest to largest. With this view, the universal coding problem is to optimally characterize types. In this paper, this Type Size approach is studied for universal source coding of an exponential family of distributions, using the most natural type class definition: two sequences are in the same type class if and only if they are indistinguishable in the sense that they have the same probability for every distribution in the family. This characterization is called the point type class. Exact third-order coding rate is derived for the resulting compression algorithm, revealing that the point type approach, while natural, is sub-optimal compared to the quantized type method, which was previously proposed by the authors.

[1]  Oliver Kosut,et al.  Third-order coding rate for universal compression of Markov sources , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[2]  Sebastian Tschiatschek,et al.  Introduction to Probabilistic Graphical Models , 2014 .

[3]  Oliver Kosut,et al.  Fine Asymptotics for Universal One-to-One Compression of Parametric Sources , 2016, IEEE Transactions on Information Theory.

[4]  Gadiel Seroussi.  On universal types , 2004, ISIT.

[5]  De-lin Ren,et al.  Topics in Integral Geometry , 1994 .

[6]  Gadiel Seroussi,et al.  Optimal Algorithms for Universal Random Number Generation From Finite Memory Sources , 2014, IEEE Transactions on Information Theory.

[7]  N. Merhav,et al.  On universal simulation of information sources using training data , 2002, Proceedings IEEE International Symposium on Information Theory,.

[8]  Oliver Kosut,et al.  A new Type Size code for universal one-to-one compression of parametric sources , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[9]  A. A. Borovkov,et al.  Integro-Local Limit Theorems Including Large Deviations for Sums of Random Vectors. II , 1999 .

[10]  Oliver Kosut,et al.  Universal fixed-to-variable source coding in the finite blocklength regime , 2013, 2013 IEEE International Symposium on Information Theory.