Kolmogorov's spline network

In this paper, an innovative neural-network architecture is proposed and elucidated. This architecture, based on the Kolmogorov's superposition theorem (1957) and called the Kolmogorov's spline network (KSN), utilizes more degrees of adaptation to data than currently used neural-network architectures (NNAs). By using cubic spline technique of approximation, both for activation and internal functions, more efficient approximation of multivariate functions can be achieved. The bound on approximation error and number of adjustable parameters, derived in this paper, favorably compares KSN with other one-hidden layer feedforward NNAs. The training of KSN, using the ensemble approach and the ensemble multinet, is described. A new explicit algorithm for constructing cubic splines is presented.

[1]  Yoh-Han Pao,et al.  The ensemble approach to neural-network learning and generalization , 1999, IEEE Trans. Neural Networks.

[2]  David A. Sprecher,et al.  A Numerical Implementation of Kolmogorov's Superpositions , 1996, Neural Networks.

[3]  Vladik Kreinovich,et al.  Guaranteed Intervals for Kolmogorov’s Theorem (and Their Possible Relation to Neural Networks) , 2004 .

[4]  Y. Makovoz.  Random Approximants and Neural Networks , 1996 .

[5]  David A. Sprecher,et al.  A Numerical Implementation of Kolmogorov's Superpositions II , 1996, Neural Networks.

[6]  Massood Tabib-Azar,et al.  A net with complex weights , 2001, IEEE Trans. Neural Networks.

[7]  Francesco Piazza,et al.  Multilayer feedforward networks with adaptive spline activation function , 1999, IEEE Trans. Neural Networks.

[8]  Francesco Piazza,et al.  Learning and Approximation Capabilities of Adaptive Spline Activation Function Neural Networks , 1998, Neural Networks.

[9]  Paolo Campolucci,et al.  Complex-valued neural networks with adaptive spline activation function for digital-radio-links nonlinear equalization , 1999, IEEE Trans. Signal Process..

[10]  Lionel Tarassenko,et al.  Estimations of error bounds for neural-network function approximators , 1999, IEEE Trans. Neural Networks.

[11]  D. Sprecher.  On the structure of continuous functions of several variables , 1965 .

[12]  M. Nees.  Approximative versions of Kolmogorov's superposition theorem, proved constructively , 1994 .

[13]  C. A. Hall,et al.  On error bounds for spline interpolation , 1968 .

[14]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[15]  J. L. Walsh,et al.  The theory of splines and their applications , 1969 .

[16]  R. Schapire.  The Strength of Weak Learnability , 1990, Machine Learning.

[17]  V. Tikhomirov.  On the Representation of Continuous Functions of Several Variables as Superpositions of Continuous Functions of one Variable and Addition , 1991 .

[18]  G. Lorentz,et al.  Constructive approximation : advanced problems , 1996 .

[19]  P. M. Prenter.  Splines and variational methods , 1975 .

[20]  A. Barron.  Approximation and Estimation Bounds for Artificial Neural Networks , 1991, COLT '91.

[21]  Vra Krkov.  Kolmogorov's Theorem Is Relevant , 1991, Neural Computation.

[22]  Vera Kurková,et al.  Kolmogorov's theorem and multilayer neural networks , 1992, Neural Networks.

[23]  Leo Breiman,et al.  Hinging hyperplanes for regression, classification, and function approximation , 1993, IEEE Trans. Inf. Theory.

[24]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[25]  Ah Chung Tsoi,et al.  Universal Approximation Using Feedforward Neural Networks: A Survey of Some Existing Methods, and Some New Results , 1998, Neural Networks.

[26]  L. K. Jones,et al.  Good weights and hyperbolic kernels for neural networks, projection pursuit, and pattern classification: Fourier strategies for extracting information from high-dimensional data , 1994, IEEE Trans. Inf. Theory.

[27]  Andrew R. Barron,et al.  Approximation and estimation bounds for artificial neural networks , 2004, Machine Learning.

[28]  Chuanyi Ji,et al.  Combinations of Weak Classifiers , 1996, NIPS.

[29]  Tomaso A. Poggio,et al.  Representation Properties of Networks: Kolmogorov's Theorem Is Irrelevant , 1989, Neural Computation.