Regularization Algorithms for Learning That Are Equivalent to Multilayer Networks

Learning an input-output mapping from a set of examples, of the type that many neural networks have been constructed to perform, can be regarded as synthesizing an approximation of a multidimensional function (that is, solving the problem of hypersurface reconstruction). From this point of view, this form of learning is closely related to classical approximation techniques, such as generalized splines and regularization theory. A theory is reported that shows the equivalence between regularization and a class of three-layer networks called regularization networks or hyper basis functions. These networks are not only equivalent to generalized splines but are also closely related to the classical radial basis functions used for interpolation tasks and to several pattern recognition and neural network algorithms. They also have an interesting interpretation in terms of prototypes that are synthesized and optimally combined during the learning stage.

[1]  R. Ouboter,et al.  The relationship between current, voltage and magnetic field for a superconducting double-point-contact diode placed in the center of a coaxial cavity , 1967 .

[2]  G. Wahba,et al.  A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines , 1970 .

[3]  F. Kárpáti,et al.  [Mast cell destruction, a new therapeutic possibility in the treatment of interstitial cystitis]. , 1971, Der Urologe.

[4]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[5]  Its Mechanism,et al.  Biological rhythms and their central mechanism , 1979 .

[6]  G. York,et al.  Congenital and Acquired Cognitive Disorders , 1981, Neurology.

[7]  L. Schumaker Spline Functions: Basic Theory , 1981 .

[8]  T. Kohonen Self-organized formation of topographically correct feature maps , 1982 .

[9]  R. Franke Scattered data interpolation: tests of some methods , 1982 .

[10]  David J. Hand,et al.  Kernel Discriminant Analysis , 1983 .

[11]  Tomaso Poggio,et al.  Computational vision and regularization theory , 1985, Nature.

[12]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[13]  Tomaso Poggio,et al.  Probabilistic Solution of Ill-Posed Problems in Computational Vision , 1987 .

[14]  A. J. Mistlin,et al.  Visual neurones responsive to faces , 1987, Trends in Neurosciences.

[15]  M. Bertero,et al.  Ill-posed problems in early vision , 1988, Proc. IEEE.

[16]  J. Keeler Comparison Between Kanerva's SDM and Hopfield-Type Neural Networks , 1988, Cogn. Sci..

[17]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[18]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[19]  David E. Rumelhart,et al.  Product Units: A Computationally Powerful and Biologically Plausible Extension to Backpropagation Networks , 1989, Neural Computation.

[20]  Martin Casdagli,et al.  Nonlinear prediction of chaotic time series , 1989 .