Approximation by superpositions of a sigmoidal function

In this paper we demonstrate that finite linear combinations of compositions of a fixed, univariate function and a set of affine functionals can uniformly approximate any continuous function ofn real variables with support in the unit hypercube; only mild conditions are imposed on the univariate function. Our results settle an open question about representability in the class of single hidden layer neural networks. In particular, we show that arbitrary decision regions can be arbitrarily well approximated by continuous feedforward neural networks with only a single internal, hidden layer and any continuous sigmoidal nonlinearity. The paper discusses approximation properties of other possible types of nonlinearities that might be implemented by artificial neural networks.

[1]  W. Rudin.  Real and complex analysis , 1968 .

[2]  L. Brown,et al.  Spectral synthesis and the Pompeiu problem , 1973 .

[3]  R. Ash,et al.  Real analysis and probability , 1975 .

[4]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[5]  P. Diaconis,et al.  On Nonlinear Functions of Linear Combinations , 1984 .

[6]  David Haussler,et al.  Classifying learnable geometric concepts with the Vapnik-Chervonenkis dimension , 1986, STOC '86.

[7]  Geoffrey E. Hinton,et al.  A general framework for parallel distributed processing , 1986 .

[8]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[9]  C. Micchelli,et al.  Some remarks on ridge functions , 1987 .

[10]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[11]  Richard Lippmann,et al.  Neural Net and Traditional Classifiers , 1987, NIPS.

[12]  B. Bavarian,et al.  Introduction to neural networks for intelligent control , 1988, IEEE Control Systems Magazine.

[13]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[14]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[15]  R. M. Dudley,et al.  Real Analysis and Probability , 1989 .

[16]  S. M. Carroll,et al.  Construction of neural nets using the radon transform , 1989, International 1989 Joint Conference on Neural Networks.

[17]  A. El-Jaroudi,et al.  Classification capabilities of two-layer neural nets , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[18]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[19]  L. Jones.  Constructive approximations for neural networks by sigmoidal functions , 1990, Proc. IEEE.

[20]  V. Tikhomirov.  On the Representation of Continuous Functions of Several Variables as Superpositions of Continuous Functions of one Variable and Addition , 1991 .