Achieving the rate-distortion bound with low-density generator matrix codes

It is shown that binary low-density generator matrix codes can achieve the rate-distortion bound of discrete memoryless sources with general distortion measure via multilevel quantization. A practical encoding scheme based on the survey-propagation algorithm is proposed. The effectiveness of the proposed scheme is verified through simulation.

[1]  M. Mézard,et al.  Analytic and Algorithmic Solution of Random Satisfiability Problems , 2002, Science.

[2]  Toby Berger,et al.  Coding for noisy channels with input-dependent insertions , 1977, IEEE Trans. Inf. Theory.

[3]  R. Gallager.  Information Theory and Reliable Communication , 1968 .

[4]  Tsachy Weissman,et al.  Rate-distortion via Markov chain Monte Carlo , 2008, 2008 IEEE International Symposium on Information Theory.

[5]  Rüdiger L. Urbanke,et al.  Lower bounds on the rate-distortion function of individual LDGM codes , 2008, 2008 5th International Symposium on Turbo Codes and Related Topics.

[6]  James L Massey.  Joint Source and Channel Coding , 1977 .

[7]  Tsachy Weissman,et al.  The empirical distribution of rate-constrained source codes , 2004, IEEE Transactions on Information Theory.

[8]  Jessica J. Fridrich,et al.  Binary quantization using Belief Propagation with decimation over factor graphs of LDGM codes , 2007, ArXiv.

[9]  Sergio Verdú,et al.  Nonlinear Sparse-Graph Codes for Lossy Compression , 2009, IEEE Transactions on Information Theory.

[10]  Tsachy Weissman,et al.  Rate-distortion in near-linear time , 2008, 2008 IEEE International Symposium on Information Theory.

[11]  Teofilo C. Ancheta.  Syndrome-source-coding and its universal generalization , 1976, IEEE Trans. Inf. Theory.

[12]  Riccardo Zecchina,et al.  Survey propagation: An algorithm for satisfiability , 2002, Random Struct. Algorithms.

[13]  Frederick Jelinek.  Tree encoding of memoryless time-discrete sources with a fidelity criterion , 1969, IEEE Trans. Inf. Theory.

[14]  Alexandros G. Dimakis,et al.  Lower bounds on the rate-distortion function of LDGM codes , 2007, 2007 IEEE Information Theory Workshop.

[15]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[16]  T. Murayama.  Thouless-Anderson-Palmer approach for lossy compression. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[17]  R. Urbanke,et al.  Polar codes are optimal for lossy source coding , 2009 .

[18]  Aaron D. Wyner,et al.  Coding Theorems for a Discrete Source With a Fidelity CriterionInstitute of Radio Engineers, International Convention Record, vol. 7, 1959. , 1993 .

[19]  Andrew J. Viterbi,et al.  Trellis Encoding of memoryless discrete-time sources with a fidelity criterion , 1974, IEEE Trans. Inf. Theory.

[20]  Martin J. Wainwright,et al.  Lossy source encoding via message-passing and decimation over generalized codewords of LDGM codes , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[21]  Emin Martinian,et al.  Iterative Quantization Using Codes On Graphs , 2004, ArXiv.

[22]  Sanjeev Khudanpur,et al.  Typicality of a Good Rate-Distortion Code , 2004 .

[23]  Martin J. Wainwright,et al.  A new look at survey propagation and its generalizations , 2004, SODA '05.

[24]  Toby Berger,et al.  Fixed-slope universal lossy data compression , 1997, IEEE Trans. Inf. Theory.

[25]  Martin J. Wainwright,et al.  Low-Density Graph Codes That Are Optimal for Binning and Coding With Side Information , 2009, IEEE Transactions on Information Theory.

[26]  Amir Dembo,et al.  Source coding, large deviations, and approximate pattern matching , 2001, IEEE Trans. Inf. Theory.

[27]  Ioannis Kontoyiannis,et al.  An implementable lossy version of the Lempel-Ziv algorithm - Part I: Optimality for memoryless sources , 1999, IEEE Trans. Inf. Theory.

[28]  En-Hui Yang,et al.  Simple universal lossy data compression schemes derived from the Lempel-Ziv algorithm , 1996, IEEE Trans. Inf. Theory.

[29]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[30]  Alexander Barg,et al.  Random codes: Minimum distances and error exponents , 2002, IEEE Trans. Inf. Theory.

[31]  Riccardo Zecchina,et al.  Lossy data compression with random gates. , 2005, Physical review letters.