Average-Case Learning Curves for Radial Basis Function Networks

The application of statistical physics to the study of the learning curves of feedforward connectionist networks has to date been concerned mostly with perceptron-like networks. Recent work has extended the theory to networks such as committee machines and parity machines, and an important direction for current and future research is the extension of this body of theory to further connectionist networks. In this article, we use this formalism to investigate the learning curves of gaussian radial basis function networks (RBFNs) having fixed basis functions. (These networks have also been called generalized linear regression models.) We address the problem of learning linear and nonlinear, realizable and unrealizable, target rules from noise-free training examples using a stochastic training algorithm. Expressions for the generalization error, defined as the expected error for a network with a given set of parameters, are derived for general gaussian RBFNs, for which all parameters, including centers and spread parameters, are adaptable. Specializing to the case of RBFNs with fixed basis functions (basis functions having parameters chosen without reference to the training examples), we then study the learning curves for these networks in the limit of high temperature.

[1]  Sompolinsky,et al.  Statistical mechanics of learning from examples. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[2]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[3]  Peter L. Bartlett,et al.  Lower bounds on the VC-dimension of smoothly parametrized function classes , 1994, COLT '94.

[4]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[5]  David S. Broomhead,et al.  Multivariable Functional Interpolation and Adaptive Networks , 1988, Complex Syst..

[6]  Martin Anthony,et al.  Computational learning theory: an introduction , 1992 .

[7]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[8]  Oh,et al.  Generalization in a two-layer neural network. , 1993, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[9]  Martin Anthony,et al.  Quantifying Generalization in Linearly Weighted Neural Networks , 1994, Complex Syst..

[10]  Hansel,et al.  Broken symmetries in multilayered perceptrons. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[11]  Peter J. W. Rayner,et al.  Generalization and PAC learning: some new results for the class of generalized single-layer networks , 1995, IEEE Trans. Neural Networks.

[12]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[13]  Simon J. Godsill,et al.  The restoration of degraded audio signals , 1993 .

[14]  David Saad,et al.  Learning and Generalization in Radial Basis Function Networks , 1995, Neural Computation.

[15]  Esther Levin,et al.  A statistical approach to learning and generalization in layered neural networks , 1989, Proc. IEEE.

[16]  Mahesan Niranjan,et al.  Neural networks and radial basis functions in classifying static speech patterns , 1990 .

[17]  Sompolinsky,et al.  Learning from examples in large neural networks. , 1990, Physical review letters.

[18]  M. J. D. Powell,et al.  Radial basis functions for multivariable interpolation: a review , 1987 .

[19]  Wj Fitzgerald,et al.  The restoration of digital audio recordings using the Gibbs sampler , 1993 .

[20]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[21]  Martin Anthony,et al.  Computational Learning Theory , 1992 .

[22]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[23]  S. Renals,et al.  Phoneme classification experiments using radial basis functions , 1989, International 1989 Joint Conference on Neural Networks.

[24]  Esther Levin,et al.  A statistical approach to learning and generalization in layered neural networks , 1989, COLT '89.

[25]  Wilfred Kaplan,et al.  Advanced mathematics for engineers , 1981 .