The Boltzmann g-RHONN: A learning machine for estimating unknown probability distributions

Abstract This paper considers the following problem: assume that we have an ergodic signal source S that each time transmits a multidimensional signal x according to an unknown ergodic probability distribution with density p(x). Then the problem is to estimate the unknown density p(x). The problem is solved via gradient recurrent high-order neural network (g-RHONN) models whose weights are adjusted according to appropriate learning laws. In the proposed method the signals are considered to be the states of a stochastic gradient dynamical system (Langevin s.d.e.) after its convergence (in a stochastic manner). Then the (unknown) system is identified (approximately) using g-RHONNs. After the learning procedure converges, the energy function of the neural network is the estimate of the logarithm of the unknown probability distribution. Extensions are also provided for estimation of unknown joint probability distributions.

[1]  Bart Kosko.  Stochastic competitive learning , 1991, IEEE Trans. Neural Networks.

[2]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[3]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[4]  B. Kosko.  Structural stability of unsupervised learning in feedback neural networks , 1991 .

[5]  Neil E. Cotter,et al.  The Stone-Weierstrass theorem and its application to neural networks , 1990, IEEE Trans. Neural Networks.

[6]  Shun-ichi Amari,et al.  Information geometry of Boltzmann machines , 1992, IEEE Trans. Neural Networks.

[7]  Jürgen Schmidhuber,et al.  Learning Factorial Codes by Predictability Minimization , 1992, Neural Computation.

[8]  Mats Bengtsson.  Stochastic optimization algorithms - an application to pattern matching , 1990, Pattern Recognit. Lett..

[9]  A. Dembo,et al.  High-order absolutely stable neural networks , 1991 .

[10]  Manolis A. Christodoulou,et al.  Identification of nonlinear systems using new dynamic neural network structures , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[11]  Lei Guo,et al.  Continuous-time stochastic adaptive tracking—robustness and asymptotic properties , 1990 .

[12]  Stephen Grossberg,et al.  Studies of mind and brain , 1982 .

[13]  Bart Kosko,et al.  Differential competitive learning for centroid estimation and phoneme recognition , 1991, IEEE Trans. Neural Networks.

[14]  E. Renshaw,et al.  STOCHASTIC DIFFERENTIAL EQUATIONS , 1974 .

[15]  Bart Kosko,et al.  Unsupervised learning in noise , 1990, International 1989 Joint Conference on Neural Networks.

[16]  S. Grossberg.  On learning and energy-entropy dependence in recurrent and nonrecurrent signed networks , 1969 .

[17]  Stephen Grossberg,et al.  Competitive Learning: From Interactive Activation to Adaptive Resonance , 1987, Cogn. Sci..

[18]  Joseph A. O'Sullivan,et al.  Representing and computing regular languages on massively parallel networks , 1991, IEEE Trans. Neural Networks.

[20]  S. Geman,et al.  Diffusions for global optimizations , 1986 .

[21]  H. D. Miller,et al.  The Theory Of Stochastic Processes , 1977, The Mathematical Gazette.

[22]  A. Skorokhod,et al.  Studies in the theory of random processes , 1966 .

[23]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[24]  Akira Namatame,et al.  Structural Connectionist Learning with Complementary Coding , 1992, Int. J. Neural Syst..

[25]  David M. Clark,et al.  A convergence theorem for Grossberg learning , 1990, Neural Networks.

[26]  Martin Hasler,et al.  Recursive neural networks for associative memory , 1990, Wiley-interscience series in systems and optimization.

[27]  Shun-ichi Amari,et al.  Dualistic geometry of the manifold of higher-order neurons , 1991, Neural Networks.

[28]  Manolis A. Christodoulou,et al.  Structural properties of gradient recurrent high-order neural networks , 1995 .

[29]  Padhraic Smyth,et al.  Rule-Based Neural Networks for Classification and Probability Estimation , 1992, Neural Computation.