Performance trade-offs in multi-processor approximate message passing

We consider large-scale linear inverse problems in Bayesian settings. Our general approach follows a recent line of work that applies the approximate message passing (AMP) framework in multi-processor (MP) computational systems by storing and processing a subset of rows of the measurement matrix along with corresponding measurements at each MP node. In each MP-AMP iteration, nodes of the MP system and its fusion center exchange lossily compressed messages pertaining to their estimates of the input. There is a trade-off between the physical costs of the reconstruction process including computation time, communication loads, and the reconstruction quality, and it is impossible to simultaneously minimize all the costs. We pose this minimization as a multi-objective optimization problem (MOP), and study the properties of the best trade-offs (Pareto optimality) in this MOP. We prove that the achievable region of this MOP is convex, and conjecture how the combined cost of computation and communication scales with the desired mean squared error. These properties are verified numerically.

[1]  Toby Berger,et al.  Rate distortion theory : a mathematical basis for data compression , 1971 .

[2]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[3]  Robert M. Gray,et al.  An Algorithm for Vector Quantizer Design , 1980, IEEE Trans. Commun..

[4]  Ruixin Niu,et al.  Multi-processor approximate message passing using lossy compression , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  Richard E. Blahut,et al.  Computation of channel capacity and rate-distortion functions , 1972, IEEE Trans. Inf. Theory.

[6]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[7]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[8]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, 2010 IEEE International Symposium on Information Theory.

[9]  Andrea Montanari,et al.  Graphical Models Concepts in Compressed Sensing , 2010, Compressed Sensing.

[10]  Suguru Arimoto,et al.  An algorithm for computing the capacity of arbitrary discrete memoryless channels , 1972, IEEE Trans. Inf. Theory.

[11]  Allen Gersho,et al.  Vector quantization and signal compression , 1991, The Kluwer international series in engineering and computer science.

[12]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[13]  Chih-Chun Wang,et al.  Multiuser Detection of Sparsely Spread CDMA , 2008, IEEE Journal on Selected Areas in Communications.

[14]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[15]  Martin Vetterli,et al.  Rate Distortion Behavior of Sparse Sources , 2012, IEEE Transactions on Information Theory.

[16]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[17]  Gregory J. Pottie,et al.  Wireless integrated network sensors , 2000, Commun. ACM.

[18]  Sergio Verdú,et al.  Randomly spread CDMA: asymptotics via statistical physics , 2005, IEEE Transactions on Information Theory.

[19]  Yonina C. Eldar,et al.  Distributed Compressed Sensing for Static and Time-Varying Networks , 2013, IEEE Transactions on Signal Processing.

[20]  João M. F. Xavier,et al.  Distributed Basis Pursuit , 2010, IEEE Transactions on Signal Processing.

[21]  Deanna Needell,et al.  Two-Part Reconstruction With Noisy-Sudocodes , 2014, IEEE Transactions on Signal Processing.

[22]  John E. Dennis,et al.  Normal-Boundary Intersection: A New Method for Generating the Pareto Surface in Nonlinear Multicriteria Optimization Problems , 1998, SIAM J. Optim..

[23]  Kenneth Rose,et al.  A mapping approach to rate-distortion computation and analysis , 1994, IEEE Trans. Inf. Theory.

[24]  Florent Krzakala,et al.  Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices , 2012, ArXiv.

[25]  R. Gray,et al.  Vector quantization , 1984, IEEE ASSP Magazine.

[26]  Enrico Magli,et al.  Distributed Iterative Thresholding for $\ell _{0}/\ell _{1}$ -Regularized Linear Inverse Problems , 2015, IEEE Transactions on Information Theory.

[27]  Ahmad Beirami,et al.  Mismatched estimation in large linear systems , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[28]  Gaurav S. Sukhatme,et al.  Connecting the Physical World with Pervasive Networks , 2002, IEEE Pervasive Comput..

[29]  Junan Zhu,et al.  Performance regions in compressed sensing from noisy measurements , 2013, 2013 47th Annual Conference on Information Sciences and Systems (CISS).

[30]  Yonina C. Eldar,et al.  Distributed approximate message passing for sparse signal recovery , 2014, 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[31]  Yonina C. Eldar,et al.  Distributed sparse signal recovery for sensor networks , 2012, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.