Multi-processor approximate message passing using lossy compression

In this paper, a communication-efficient multi-processor compressed sensing framework based on the approximate message passing algorithm is proposed. We perform lossy compression on the data being communicated between processors, resulting in a reduction in communication costs with a minor degradation in recovery quality. In the proposed framework, a new state evolution formulation takes the quantization error into account, and analytically determines the coding rate required in each iteration. Two approaches for allocating the coding rate, an online back-tracking heuristic and an optimal allocation scheme based on dynamic programming, provide significant reductions in communication costs.

[1]  Richard E. Blahut,et al.  Computation of channel capacity and rate-distortion functions , 1972, IEEE Trans. Inf. Theory.

[2]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[3]  Kenneth Rose,et al.  A mapping approach to rate-distortion computation and analysis , 1994, IEEE Trans. Inf. Theory.

[4]  Ahmad Beirami,et al.  Mismatched estimation in large linear systems , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[5]  Gaurav S. Sukhatme,et al.  Connecting the Physical World with Pervasive Networks , 2002, IEEE Pervasive Comput..

[6]  Sergio Verdú,et al.  Randomly spread CDMA: asymptotics via statistical physics , 2005, IEEE Transactions on Information Theory.

[7]  Yonina C. Eldar,et al.  Modified distributed iterative hard thresholding , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[8]  Rüdiger L. Urbanke,et al.  Polar Codes are Optimal for Lossy Source Coding , 2009, IEEE Transactions on Information Theory.

[9]  Andrea Montanari,et al.  Graphical Models Concepts in Compressed Sensing , 2010, Compressed Sensing.

[10]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[11]  Gregory J. Pottie,et al.  Wireless integrated network sensors , 2000, Commun. ACM.

[12]  Chih-Chun Wang,et al.  Multiuser Detection of Sparsely Spread CDMA , 2008, IEEE Journal on Selected Areas in Communications.

[13]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[14]  Suguru Arimoto,et al.  An algorithm for computing the capacity of arbitrary discrete memoryless channels , 1972, IEEE Trans. Inf. Theory.

[15]  Yanting Ma,et al.  Compressive imaging via approximate message passing with wavelet-based image denoising , 2014, 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[16]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, 2010 IEEE International Symposium on Information Theory.

[17]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[18]  João M. F. Xavier,et al.  Distributed Basis Pursuit , 2010, IEEE Transactions on Signal Processing.

[19]  Volkan Cevher,et al.  Bilinear Generalized Approximate Message Passing—Part I: Derivation , 2013, IEEE Transactions on Signal Processing.

[20]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[21]  Robert M. Gray,et al.  An Algorithm for Vector Quantizer Design , 1980, IEEE Trans. Commun..

[22]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[23]  Toby Berger,et al.  Rate distortion theory : a mathematical basis for data compression , 1971 .

[24]  Bernard Widrow,et al.  Quantization Noise: Roundoff Error in Digital Computation, Signal Processing, Control, and Communications , 2008 .

[25]  Florent Krzakala,et al.  Statistical physics-based reconstruction in compressed sensing , 2011, ArXiv.

[26]  Volkan Cevher,et al.  Bilinear Generalized Approximate Message Passing , 2013, ArXiv.

[27]  John E. Dennis,et al.  Normal-Boundary Intersection: A New Method for Generating the Pareto Surface in Nonlinear Multicriteria Optimization Problems , 1998, SIAM J. Optim..

[28]  Allen Gersho,et al.  Vector quantization and signal compression , 1991, The Kluwer international series in engineering and computer science.

[29]  Andrea Montanari,et al.  The Noise-Sensitivity Phase Transition in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[30]  Dongning Guo,et al.  Asymptotic Mean-Square Optimality of Belief Propagation for Sparse Linear Systems , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Chengdu.

[31]  R. Gray,et al.  Vector quantization , 1984, IEEE ASSP Magazine.

[32]  Sundeep Rangan,et al.  Hybrid Approximate Message Passing with Applications to Structured Sparsity , 2011, ArXiv.

[33]  Florent Krzakala,et al.  Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices , 2012, ArXiv.

[34]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[35]  João M. F. Xavier,et al.  D-ADMM: A Communication-Efficient Distributed Algorithm for Separable Optimization , 2012, IEEE Transactions on Signal Processing.

[36]  Yonina C. Eldar,et al.  Distributed Compressed Sensing for Static and Time-Varying Networks , 2013, IEEE Transactions on Signal Processing.

[37]  Andrea Montanari,et al.  Non-Negative Principal Component Analysis: Message Passing Algorithms and Sharp Asymptotics , 2014, IEEE Transactions on Information Theory.

[38]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[39]  Yanting Ma,et al.  Compressive Imaging via Approximate Message Passing With Image Denoising , 2014, IEEE Transactions on Signal Processing.

[40]  Dongning Guo,et al.  A single-letter characterization of optimal noisy compressed sensing , 2009, 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[41]  Junan Zhu,et al.  Performance regions in compressed sensing from noisy measurements , 2013, 2013 47th Annual Conference on Information Sciences and Systems (CISS).

[42]  Yonina C. Eldar,et al.  Distributed approximate message passing for sparse signal recovery , 2014, 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[43]  Yonina C. Eldar,et al.  Distributed sparse signal recovery for sensor networks , 2012, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.