A BP-Based Algorithm for Performing Bayesian Inference in Large Perceptron-Type Networks

Although the Bayesian approach provides optimal performance for many inference problems, the computation cost is sometimes impractical. We herein develop a practical algorithm by which to approximate Bayesian inference in large single-layer feed-forward networks (perceptrons) based on belief propagation (BP). Although direct application of BP to the inference problem remains computationally difficult, by introducing methods and concepts from statistical mechanics that are related to the central limit theorem and the law of large numbers, the proposed BP-based algorithm exhibits nearly optimal performance in a practical time scale for ideal large networks. In order to demonstrate the practical significance of the proposed algorithm, an application to a problem that arises in a mobile communications system is also presented.

[1]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[2]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[3]  M. Opper,et al.  Tractable approximations for probabilistic models: the adaptive Thouless-Anderson-Palmer mean field approach. , 2001, Physical review letters.

[4]  Yoshiyuki Kabashima,et al.  Belief propagation vs. TAP for decoding corrupted messages , 1998 .

[5]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[6]  Tatsuto Murayama,et al.  Statistical mechanics of the data compression theorem , 2002 .

[7]  Opper,et al.  Mean field approach to Bayes learning in feed-forward neural networks. , 1996, Physical review letters.

[8]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[9]  David J. C. MacKay,et al.  Good Error-Correcting Codes Based on Very Sparse Matrices , 1997, IEEE Trans. Inf. Theory.

[10]  Mahesh K. Varanasi,et al.  Near-optimum detection in synchronous code-division multiple-access systems , 1991, IEEE Trans. Commun..

[11]  Ole Winther,et al.  A Mean Field Algorithm for Bayes Learning in Large Feed-forward Neural Networks , 1996, NIPS.

[12]  Hilbert J. Kappen,et al.  Efficient Learning in Boltzmann Machines Using Linear Response Theory , 1998, Neural Computation.

[13]  Y. Kabashima.  A CDMA multiuser detection algorithm on the basis of belief propagation , 2003 .

[14]  Kazuyuki Tanaka,et al.  Probabilistic Inference by means of Cluster Variation Method and Linear Response Theory , 2003 .

[15]  H. Nishimori.  Statistical Physics of Spin Glasses and Information Processing , 2001 .

[16]  Radford M. Neal,et al.  Near Shannon limit performance of low density parity check codes , 1996 .

[17]  Yee Whye Teh,et al.  Linear Response Algorithms for Approximate Inference in Graphical Models , 2004, Neural Computation.

[18]  Y. Iba.  The Nishimori line and Bayesian statistics , 1998, cond-mat/9809190.

[19]  Yoshiyuki Kabashima,et al.  Statistical physics of irregular low-density parity-check codes , 2000 .

[20]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[21]  Masato Okada,et al.  One step RSB scheme for the rate distortion function , 2002, cond-mat/0207637.