Lossy Source Compression Using Low-Density Generator Matrix Codes: Analysis and Algorithms

We study the use of low-density generator matrix (LDGM) codes for lossy compression of the Bernoulli symmetric source. First, we establish rigorous upper bounds on the average distortion achieved by check-regular ensemble of LDGM codes under optimal minimum distance source encoding. These bounds establish that the average distortion using such bounded degree families rapidly approaches the Shannon limit as the degrees are increased. Second, we propose a family of message-passing algorithms, ranging from the standard belief propagation algorithm at one extreme to a variant of survey propagation algorithm at the other. When combined with a decimation subroutine and applied to LDGM codes with suitably irregular degree distributions, we show that such a message-passing/decimation algorithm yields distortion very close to the Shannon rate-distortion bound for the binary symmetric source.

[1]  T. Murayama.  Thouless-Anderson-Palmer approach for lossy compression. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[2]  T. J. Goblick,et al.  Coding for a discrete information source with a distortion measure , 1963 .

[3]  Daniel A. Spielman,et al.  Improved low-density parity-check codes using irregular graphs and belief propagation , 1998, Proceedings. 1998 IEEE International Symposium on Information Theory (Cat. No.98CH36252).

[4]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[5]  H.-A. Loeliger,et al.  An introduction to factor graphs , 2004, IEEE Signal Process. Mag..

[6]  Robert Michael Tanner,et al.  A recursive approach to low complexity codes , 1981, IEEE Trans. Inf. Theory.

[7]  Nadia Creignou,et al.  Approximating The Satisfiability Threshold For Random K-Xor-Formulas , 2003, Comb. Probab. Comput..

[8]  Riccardo Zecchina,et al.  Alternative solutions to diluted p-spin models and XORSAT problems , 2002, ArXiv.

[9]  Martin J. Wainwright,et al.  Low-density constructions can achieve the Wyner-Ziv and Gelfand-Pinsker bounds , 2006, 2006 IEEE International Symposium on Information Theory.

[10]  Rajeev Motwani,et al.  Randomized algorithms , 1996, CSUR.

[11]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[12]  Emin Martinian,et al.  Iterative Quantization Using Codes On Graphs , 2004, ArXiv.

[13]  M. Mézard,et al.  Analytic and Algorithmic Solution of Random Satisfiability Problems , 2002, Science.

[14]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[15]  Jessica J. Fridrich,et al.  Practical methods for minimizing embedding impact in steganography , 2007, Electronic Imaging.

[16]  Noga Alon,et al.  The Probabilistic Method, Second Edition , 2004 .

[17]  Martin J. Wainwright,et al.  Low density codes achieve the rate-distortion bound , 2006, Data Compression Conference (DCC'06).

[18]  Martin J. Wainwright,et al.  A new look at survey propagation and its generalizations , 2004, SODA '05.

[19]  Martin J. Wainwright,et al.  Lossy source encoding via message-passing and decimation over generalized codewords of LDGM codes , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[20]  Axthonv G. Oettinger,et al.  IEEE Transactions on Information Theory , 1998 .

[21]  Rüdiger L. Urbanke,et al.  Design of capacity-approaching irregular low-density parity-check codes , 2001, IEEE Trans. Inf. Theory.

[22]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems , 1988 .

[23]  Alain Glavieux,et al.  Reflections on the Prize Paper : "Near optimum error-correcting coding and decoding: turbo codes" , 1998 .

[24]  Daniel A. Spielman,et al.  Efficient erasure correcting codes , 2001, IEEE Trans. Inf. Theory.

[25]  Olivier Dubois,et al.  The 3-XORSAT threshold , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[26]  Marc Mézard,et al.  The theoretical capacity of the Parity Source Coder , 2005, ArXiv.

[27]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[28]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[29]  Hirosuke Yamamoto,et al.  A coding theorem for lossy data compression by LDPC codes , 2003, IEEE Trans. Inf. Theory.

[30]  Michael W. Marcellin,et al.  Trellis coded quantization of memoryless and Gauss-Markov sources , 1990, IEEE Trans. Commun..

[31]  Dimitris Achlioptas,et al.  THE THRESHOLD FOR RANDOM k-SAT IS 2k log 2 O(k) , 2004, FOCS 2004.

[32]  Martin J. Wainwright,et al.  Analysis of LDGM and compound codes for lossy compression and binning , 2006, ArXiv.

[33]  Martin J. Wainwright,et al.  Low-Density Graph Codes That Are Optimal for Binning and Coding With Side Information , 2009, IEEE Transactions on Information Theory.

[34]  M. Mézard,et al.  Two Solutions to Diluted p-Spin Models and XORSAT Problems , 2003 .

[35]  Ying Zhao,et al.  Compression of binary memoryless sources using punctured turbo codes , 2002, IEEE Communications Letters.

[36]  Andrew J. Viterbi,et al.  Trellis Encoding of memoryless discrete-time sources with a fidelity criterion , 1974, IEEE Trans. Inf. Theory.

[37]  Gregory W. Wornell,et al.  Information Embedding Codes on Graphs with Iterative Encoding and Decoding , 2006, 2006 IEEE International Symposium on Information Theory.

[38]  Noga Alon,et al.  The Probabilistic Method , 2015, Fundamentals of Ramsey Theory.

[39]  Masato Okada,et al.  One step RSB scheme for the rate distortion function , 2002, cond-mat/0207637.

[40]  G. Grimmett,et al.  Probability and random processes , 2002 .

[41]  Sae-Young Chung,et al.  On the design of low-density parity-check codes within 0.0045 dB of the Shannon limit , 2001, IEEE Communications Letters.

[42]  Martin J. Wainwright,et al.  Using linear programming to Decode Binary linear codes , 2005, IEEE Transactions on Information Theory.

[43]  Cesk,et al.  MINIMIZING EMBEDDING IMPACT IN STEGANOGRAPHY USING LOW DENSITY CODES , 2007 .

[44]  Rüdiger L. Urbanke,et al.  Modern Coding Theory , 2008 .

[45]  M. Mézard,et al.  Random K-satisfiability problem: from an analytic solution to an efficient algorithm. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[46]  Alexandros G. Dimakis,et al.  Lower bounds on the rate-distortion function of LDGM codes , 2007, 2007 IEEE Information Theory Workshop.

[47]  R. Monasson,et al.  Rigorous decimation-based construction of ground pure states for spin-glass models on random lattices. , 2002, Physical review letters.

[48]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[49]  Rüdiger L. Urbanke,et al.  Lower bounds on the rate-distortion function of individual LDGM codes , 2008, 2008 5th International Symposium on Turbo Codes and Related Topics.

[50]  R. Zecchina,et al.  Message-Passing Algorithms for Non-Linear Nodes and Data Compression , 2005, Complexus.

[51]  Jessica J. Fridrich,et al.  Binary quantization using Belief Propagation with decimation over factor graphs of LDGM codes , 2007, ArXiv.

[52]  Sergio Verdú,et al.  Nonlinear Sparse-Graph Codes for Lossy Compression , 2009, IEEE Transactions on Information Theory.