Multi-layer generalized linear estimation

We consider the problem of reconstructing a signal from multi-layered (possibly) non-linear measurements. Using non-rigorous but standard methods from statistical physics we present the Multi-Layer Approximate Message Passing (ML-AMP) algorithm for computing marginal probabilities of the corresponding estimation problem and derive the associated state evolution equations to analyze its performance. We also give the expression of the asymptotic free energy and the minimal information-theoretically achievable reconstruction error. Finally, we present some applications of this measurement model for compressed sensing and perceptron learning with structured matrices/patterns, and for a simple model of estimation of latent variables in an auto-encoder.

[1]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[2]  Jukka-Pekka Onnela,et al.  Community Structure in Time-Dependent, Multiscale, and Multiplex Networks , 2009, Science.

[3]  Sundeep Rangan,et al.  Generalized approximate message passing for estimation with random linear mixing , 2010, 2011 IEEE International Symposium on Information Theory Proceedings.

[4]  Yoshiyuki Kabashima,et al.  Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels , 2007, ArXiv.

[5]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[6]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[7]  Sundeep Rangan,et al.  Hybrid generalized approximate message passing with applications to structured sparsity , 2012, 2012 IEEE International Symposium on Information Theory Proceedings.

[8]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[9]  M. Mézard.  The space of interactions in neural networks: Gardner's computation with the cavity method , 1989 .

[10]  B. Henri.  Aalborg Universitet SAMP : Approximate Message Passing for General Matrix Ensembles , 2014 .

[11]  Mikko Vehkaperä,et al.  Signal recovery using expectation consistent approximation for linear observations , 2014, 2014 IEEE International Symposium on Information Theory.

[12]  Sebastian Fischer,et al.  Exploring Artificial Intelligence In The New Millennium , 2016 .

[13]  E. Gardner,et al.  Three unfinished works on the optimal storage capacity of networks , 1989 .

[14]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, 2010 IEEE International Symposium on Information Theory.

[15]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[16]  Ole Winther,et al.  S-AMP: Approximate message passing for general matrix ensembles , 2014, 2014 IEEE Information Theory Workshop (ITW 2014).

[17]  William T. Freeman,et al.  Understanding belief propagation and its generalizations , 2003 .

[18]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[19]  Koujin Takeda,et al.  Analysis of CDMA systems that are characterized by eigenvalue spectrum , 2006, ArXiv.

[20]  Galen Reeves,et al.  The replica-symmetric prediction for compressed sensing with Gaussian matrices is exact , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[21]  S. Kak.  Information, physics, and computation , 1996 .

[22]  Florent Krzakala,et al.  Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices , 2012, ArXiv.

[23]  David M. Blei,et al.  Deep Exponential Families , 2014, AISTATS.

[24]  Christian Van den Broeck,et al.  Statistical Mechanics of Learning , 2001 .

[25]  Dustin Tran,et al.  Hierarchical Variational Models , 2015, ICML.

[26]  Sundeep Rangan,et al.  Vector approximate message passing , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[27]  M. Mézard.  Mean-field message-passing equations in the Hopfield model and its generalizations. , 2016, Physical review. E.

[28]  Sergio Verdú,et al.  Optimal Phase Transitions in Compressed Sensing , 2011, IEEE Transactions on Information Theory.

[29]  Nicolas Macris,et al.  The mutual information in random linear estimation , 2016, 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[30]  Shlomo Shamai,et al.  Support Recovery With Sparsely Sampled Free Random Matrices , 2011, IEEE Transactions on Information Theory.

[31]  Antonia Maria Tulino,et al.  Random Matrix Theory and Wireless Communications , 2004, Found. Trends Commun. Inf. Theory.