Online Learning in Radial Basis Function Networks

An analytic investigation of the average case learning and generalization properties of radial basis function (RBFs) networks is presented, utilizing online gradient descent as the learning rule. The analytic method employed allows both the calculation of generalization error and the examination of the internal dynamics of the network. The generalization error and internal dynamics are then used to examine the role of the learning rate and the specialization of the hidden units, which gives insight into decreasing the time required for training. The realizable and some over realizable cases are studied in detail: the phase of learning in which the hidden units are unspecialized (symmetric phase) and the phase in which asymptotic convergence occurs are analyzed, and their typical properties found. Finally, simulations are performed that strongly confirm the analytic results.

[1]  D. Saad,et al.  Dynamics of on-line learning in radial basis function networks , 1997 .

[2]  Heskes,et al.  Learning processes in neural networks. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[3]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[4]  H. Schwarze.  Learning a rule in a multilayer neural network , 1993 .

[5]  Todd K. Leen,et al.  Optimal Stochastic Search and Adaptive Momentum , 1993, NIPS.

[6]  Michael Biehl,et al.  Learning by on-line gradient descent , 1995 .

[7]  David Saad,et al.  Learning and Generalization in Radial Basis Function Networks , 1995, Neural Computation.

[8]  J. Andrew Ware,et al.  Layered Neural Networks as Universal Approximators , 1997, Fuzzy Days.

[9]  David Haussler,et al.  The Probably Approximately Correct (PAC) and Other Learning Models , 1993 .

[10]  Saad,et al.  Exact solution for on-line learning in multilayer neural networks. , 1995, Physical review letters.

[11]  Michael Biehl,et al.  On-line backpropagation in two-layered neural networks , 1995 .

[12]  David Barber,et al.  Finite-size effects in on-line learning of multilayer neural networks , 1996 .

[13]  Federico Girosi,et al.  On the Relationship between Generalization Error, Hypothesis Complexity, and Sample Complexity for Radial Basis Functions , 1996, Neural Computation.

[14]  Peter J. W. Rayner,et al.  Generalization and PAC learning: some new results for the class of generalized single-layer networks , 1995, IEEE Trans. Neural Networks.

[15]  Shun-ichi Amari,et al.  Backpropagation and stochastic gradient descent method , 1993, Neurocomputing.

[16]  Jason A. S. Freeman,et al.  Learning and generalisation in radial basis function networks , 1997 .

[17]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[18]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[19]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[20]  Saad,et al.  On-line learning in soft committee machines. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.