On-line learning in RBF neural networks: a stochastic approach

The on-line learning of Radial Basis Function neural networks (RBFNs) is analyzed. Our approach makes use of a master equation that describes the dynamics of the weight space probability density. An approximate solution of the master equation is obtained in the limit of a small learning rate. In this limit, the on line learning dynamics is analyzed and it is shown that, since fluctuations are small, dynamics can be well described in terms of evolution of the mean. This allows us to analyze the learning process of RBFNs in which the number of hidden nodes K is larger than the typically small number of input nodes N. The work represents a complementary analysis of on-line RBFNs, with respect to the previous works (Phys. Rev. E 56 (1997a) 907; Neur. Comput. 9 (1997) 1601), in which RBFNs with N >> K have been analyzed. The generalization error equation and the equations of motion of the weights are derived for generic RBF architectures, and numerically integrated in specific cases. Analytical results are then confirmed by numerical simulations. Unlike the case of large N > K we find that the dynamics in the case N < K is not affected by the problems of symmetric phases and subsequent symmetry breaking.

[1]  Heskes,et al.  Learning processes in neural networks. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[2]  Mahesan Niranjan,et al.  Average-Case Learning Curves for Radial Basis Function Networks , 1997, Neural Computation.

[3]  Saad,et al.  Exact solution for on-line learning in multilayer neural networks. , 1995, Physical review letters.

[4]  David Saad,et al.  Online Learning in Radial Basis Function Networks , 1997, Neural Computation.

[5]  David Saad,et al.  Learning and Generalization in Radial Basis Function Networks , 1995, Neural Computation.

[6]  D. Sherrington.  Stochastic Processes in Physics and Chemistry , 1983 .

[7]  D. Bedeaux,et al.  On the Relation between Master Equations and Random Walks and Their Solutions , 1971 .

[8]  T. Heskes.  On Fokker-Planck approximations of on-line learning processes , 1994 .

[9]  D. Saad,et al.  Dynamics of on-line learning in radial basis function networks , 1997 .

[10]  Michael Biehl,et al.  Statistical Mechanics of On{line Learning and Generalization the Handbook of Brain Theory and Neural Networks , 2003 .

[11]  Saad,et al.  On-line learning in soft committee machines. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[12]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[13]  N. Kampen,et al.  Stochastic processes in physics and chemistry , 1981 .

[14]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.