Comparing fixed and variable-width Gaussian networks

The role of width of Gaussians in two types of computational models is investigated: Gaussian radial-basis-functions (RBFs) where both widths and centers vary and Gaussian kernel networks which have fixed widths but varying centers. The effect of width on functional equivalence, universal approximation property, and form of norms in reproducing kernel Hilbert spaces (RKHS) is explored. It is proven that if two Gaussian RBF networks have the same input-output functions, then they must have the same numbers of units with the same centers and widths. Further, it is shown that while sets of input-output functions of Gaussian kernel networks with two different widths are disjoint, each such set is large enough to be a universal approximator. Embedding of RKHSs induced by "flatter" Gaussians into RKHSs induced by "sharper" Gaussians is described and growth of the ratios of norms on these spaces with increasing input dimension is estimated. Finally, large sets of argminima of error functionals in sets of input-output functions of Gaussian RBFs are described.

[1]  Héctor J. Sussmann,et al.  Uniqueness of the weights for minimal feedforward nets with a given input-output map , 1992, Neural Networks.

[2]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[3]  Hrushikesh Narhar Mhaskar,et al.  When is approximation by Gaussian networks necessarily a linear process? , 2004, Neural Networks.

[4]  Michael Schmitt,et al.  Descartes' Rule of Signs for Radial Basis Function Neural Networks , 2002, Neural Computation.

[5]  Jooyoung Park,et al.  Approximation and Radial-Basis-Function Networks , 1993, Neural Computation.

[6]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[7]  Martin Burger,et al.  Error Bounds for Approximation with Neural Networks , 2001, J. Approx. Theory.

[8]  Probabilit.  Aggregation of SVM Classifiers Using Sobolev Spaces , 2008 .

[9]  V. Kůrková.  Universality and Complexity of Approximation of Multivariable Functions by Feedforward Networks , 2002 .

[10]  Marcello Sanguineti,et al.  Complexity of Gaussian-radial-basis networks approximating smooth functions , 2009, J. Complex..

[11]  Michel Verleysen,et al.  Learning in RBF networks , 1996 .

[12]  Roman Neruda,et al.  Uniqueness of Functional Representations by Gaussian Basis Function Networks , 1994 .

[13]  Alberto Tesi,et al.  On the Problem of Local Minima in Backpropagation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  T. Poggio,et al.  The Mathematics of Learning: Dealing with Data , 2005, 2005 International Conference on Neural Networks and Brain.

[15]  Paul C. Kainen,et al.  A Sobolev-type upper bound for rates of approximation by linear combinations of Heaviside plane waves , 2007, J. Approx. Theory.

[16]  Vera Kilrkova.  Universality and Complexity of Approximation of Multivariable Functions by Feedforward Networks , 2002 .

[17]  Paolo Frasconi,et al.  Learning without local minima in radial basis function networks , 1995, IEEE Trans. Neural Networks.

[18]  Federico Girosi,et al.  Regularization Theory, Radial Basis Functions and Networks , 1994 .

[19]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[20]  D. Lambie,et al.  Modern Analysis , 1966, Nature.

[21]  A. Pietsch.  Eigenvalues and S-Numbers , 1987 .

[22]  I. Omiaj,et al.  Extensions of a Theory of Networks for Approximation and Learning : dimensionality reduction and clustering , 2022 .

[23]  Paul C. Kainen,et al.  Functionally Equivalent Feedforward Neural Networks , 1994, Neural Computation.

[24]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[25]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[26]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[27]  Eduardo Sontag,et al.  For neural networks, function determines form , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[28]  Federico Girosi,et al.  An Equivalence Between Sparse Approximation and Support Vector Machines , 1998, Neural Computation.

[29]  Michel Verleysen,et al.  Width optimization of the Gaussian kernels in Radial Basis Function Networks , 2002, ESANN.

[30]  P. C. Kainen,et al.  Singularities of finite scaling functions , 1996 .

[31]  Stefanos D. Kollias,et al.  Intelligent initialization of resource allocating RBF networks , 2005, Neural Networks.

[32]  J. W. Brown,et al.  Complex Variables and Applications , 1985 .

[33]  T Poggio,et al.  Regularization Algorithms for Learning That Are Equivalent to Multilayer Networks , 1990, Science.

[34]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[35]  Chuen-Tsai Sun,et al.  Functional equivalence between radial basis function networks and fuzzy inference systems , 1993, IEEE Trans. Neural Networks.