Online Learning Dynamics of Radial Basis Function Neural Networks near the Singularity

It has been found that strange behaviours will happen because of the singularity in the parameter space (or neuro-manifold) of hierarchical models such as feed-forward neural networks, and the learning dynamics of multilayer perceptrons near the singularity has been well discussed. In this paper, the online learning dynamics near the singularity is investigated for radial basis function (RBF) neural networks with all its unit centers, widths and output weights being continuously modified using standard gradient descent algorithm. Results show that in the case of the teacher is on the singularity, if we initiate the learning process near the singularity, then the final parameter values of hidden units are dependant on their initial values: if two hidden units are initialized with similar unit centres and widths, they will overlap; otherwise, one of the hidden units will eliminate.

[1]  Shun-ichi Amari,et al.  Singularities Affect Dynamics of Learning in Neuromanifolds , 2006, Neural Comput..

[2]  Michael Biehl,et al.  Transient dynamics of on-line learning in two-layered neural networks , 1996 .

[3]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[4]  Hyeyoung Park,et al.  On-Line Learning Theory of Soft Committee Machines with Correlated Hidden Units : Steepest Gradient Descent and Natural Gradient Descent , 2002, cond-mat/0212006.

[5]  Michael Biehl,et al.  Statistical Mechanics of On{line Learning and Generalization the Handbook of Brain Theory and Neural Networks , 2003 .

[6]  Hyeyoung Park,et al.  Slow Dynamics Due to Singularities of Hierarchical Learning Machines , 2005 .

[7]  Masato Okada,et al.  On-Line Learning Dynamics of Multilayer Perceptrons with Unidentifiable Parameters , 2003 .

[8]  D. Saad,et al.  Dynamics of on-line learning in radial basis function networks , 1997 .

[9]  Federico Girosi,et al.  On the Relationship between Generalization Error, Hypothesis Complexity, and Sample Complexity for Radial Basis Functions , 1996, Neural Computation.

[10]  Michael Biehl,et al.  On-line backpropagation in two-layered neural networks , 1995 .

[11]  Silvia Scarpetta,et al.  On-line learning in RBF neural networks: a stochastic approach , 2000, Neural Networks.

[12]  S. Amari,et al.  Dynamics of Learning in Multilayer Perceptrons , 2008 .

[13]  Saad,et al.  Exact solution for on-line learning in multilayer neural networks. , 1995, Physical review letters.

[14]  Saad,et al.  On-line learning in soft committee machines. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[15]  Hyeyoung Park,et al.  Dynamics of the adaptive natural gradient descent method for soft committee machines. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[16]  David Saad,et al.  Online Learning in Radial Basis Function Networks , 1997, Neural Computation.