Second-order properties of lossy likelihoods and the MLE/MDL dichotomy in lossy compression

This paper develops a theoretical framework for lossy source coding that treats it as a statistical problem, in analogy to the approach to universal lossless coding suggested by Rissanen’s Minimum Description Length (MDL) principle. Two methods for selecting efficient compression algorithms are proposed, based on lossy variants of the Maximum Likelihood and MDL principles. Their theoretical performance is analyzed, and it is shown under appropriate assumptions that the MDL approach to universal lossy coding identifies the optimal model class of lossy codes.

[1]  Jorma Rissanen,et al.  The Minimum Description Length Principle in Coding and Modeling , 1998, IEEE Trans. Inf. Theory.

[2]  Matthew T. Harrison.  The Convergence of Lossy Maximum Likelihood Estimators , 2003 .

[3]  En-Hui Yang,et al.  On the Performance of Data Compression Algorithms Based Upon String Matching , 1998, IEEE Trans. Inf. Theory.

[4]  Wojciech Szpankowski,et al.  A suboptimal lossy data compression based on approximate pattern matching , 1997, IEEE Trans. Inf. Theory.

[5]  Leon Gordon Kraft,et al.  A device for quantizing, grouping, and coding amplitude-modulated pulses , 1949 .

[6]  Junshan Zhang,et al.  Arbitrary source models and Bayesian codebooks in rate-distortion theory , 2002, IEEE Trans. Inf. Theory.

[7]  Ioannis Kontoyiannis,et al.  An implementable lossy version of the Lempel-Ziv algorithm - Part I: Optimality for memoryless sources , 1999, IEEE Trans. Inf. Theory.

[8]  Aaron D. Wyner,et al.  Coding Theorems for a Discrete Source With a Fidelity CriterionInstitute of Radio Engineers, International Convention Record, vol. 7, 1959. , 1993 .

[9]  J. Berning.  On the Multivariate Law of the Iterated Logarithm , 1979 .

[10]  Benedikt M. Pötscher,et al.  A UNIFORM LAW OF LARGE NUMBERS FOR DEPENDENT AND HETEROGENEOUS DATA PROCESSES , 1989 .

[11]  D. Andrews.  CONSISTENCY IN NONLINEAR ECONOMETRIC MODELS: A GENERIC UNIFORM LAW OF LARGE NUMBERS , 1987 .

[12]  Gerald Matz,et al.  Information geometric formulation and interpretation of accelerated Blahut-Arimoto-type algorithms , 2004, Information Theory Workshop.

[13]  John C. Kieffer,et al.  Sample converses in source coding theory , 1991, IEEE Trans. Inf. Theory.

[14]  A. Dembo,et al.  The asymptotics of waiting times between stationary processes , 1999 .

[15]  Zhen Zhang,et al.  On the redundancy of trellis lossy source coding , 2002, IEEE Trans. Inf. Theory.

[16]  Zhen Zhang,et al.  On the Redundancy of Lossy Source Coding with Abstract Alphabets , 1999, IEEE Trans. Inf. Theory.

[17]  En-Hui Yang,et al.  The redundancy of source coding with a fidelity criterion - Part II: Coding at a fixed rate level with unknown statistics , 2001, IEEE Trans. Inf. Theory.

[18]  Ioannis Kontoyiannis,et al.  Pointwise redundancy in lossy data compression and universal lossy data compression , 2000, IEEE Trans. Inf. Theory.

[19]  Amir Dembo,et al.  Source coding, large deviations, and approximate pattern matching , 2001, IEEE Trans. Inf. Theory.

[20]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[21]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[22]  Kenneth Rose,et al.  Natural type selection in adaptive lossy compression , 2001, IEEE Trans. Inf. Theory.

[23]  Zhen Zhang,et al.  The redundancy of source coding with a fidelity criterion: 1. Known statistics , 1997, IEEE Trans. Inf. Theory.

[24]  Brockway McMillan,et al.  Two inequalities implied by unique decipherability , 1956, IRE Trans. Inf. Theory.

[25]  Yuehua Wu,et al.  Model selection with data-oriented penalty , 1999 .

[26]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[27]  Yossef Steinberg,et al.  An algorithm for source coding subject to a fidelity criterion, based on string matching , 1993, IEEE Trans. Inf. Theory.

[28]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[29]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[30]  The First Order Asymptotics of Waiting Times between Stationary Processes under Nonstandard Conditions , 2003 .

[31]  Andrew R. Barron,et al.  Minimum complexity density estimation , 1991, IEEE Trans. Inf. Theory.