Almost optimal estimates for approximation and learning by radial basis function networks

This paper quantifies the approximation capability of radial basis function networks (RBFNs) and their applications in machine learning theory. The target is to deduce almost optimal rates of approximation and learning by RBFNs. For approximation, we show that for large classes of functions, the convergence rate of approximation by RBFNs is not slower than that of multivariate algebraic polynomials. For learning, we prove that, using the classical empirical risk minimization, the RBFNs estimator can theoretically realize the almost optimal learning rate. The obtained results underlie the successful application of RBFNs in various machine learning problems.

[1]  Simon Hubbert,et al.  Radial basis functions for the sphere , 2015 .

[2]  Zongben Xu,et al.  Essential rate for approximation by spherical neural networks , 2011, Neural Networks.

[3]  A. Bejancu,et al.  ON THE ACCURACY OF SURFACE SPLINE APPROXIMATION AND INTERPOLATION TO BUMP FUNCTIONS , 2001, Proceedings of the Edinburgh Mathematical Society.

[4]  Ronald A. DeVore,et al.  Approximation Methods for Supervised Learning , 2006, Found. Comput. Math..

[5]  Robert Schaback,et al.  Error estimates and condition numbers for radial basis function interpolation , 1995, Adv. Comput. Math..

[6]  A. Caponnetto,et al.  Optimal Rates for the Regularized Least-Squares Algorithm , 2007, Found. Comput. Math..

[7]  E. Kansa,et al.  Improved multiquadric method for elliptic partial differential equations via PDE collocation on the boundary , 2002 .

[8]  Hrushikesh Narhar Mhaskar,et al.  On the tractability of multivariate integration and approximation by neural networks , 2004, J. Complex..

[9]  Natasha Flyer,et al.  A radial basis function method for the shallow water equations on a sphere , 2009, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[10]  Michael J. Johnson.  A Bound on the Approximation Order of Surface Splines , 1998 .

[11]  Vitaly Maiorov,et al.  Approximation by neural networks and learning theory , 2006, J. Complex..

[12]  S. Mendelson,et al.  Entropy and the combinatorial dimension , 2002, math/0203275.

[13]  Felipe Cucker,et al.  Best Choices for Regularization Parameters in Learning Theory: On the Bias—Variance Problem , 2002, Found. Comput. Math..

[14]  Tony Jebara,et al.  Ellipsoidal Machines , 2007, AISTATS.

[15]  A. Barron,et al.  Approximation and learning by greedy algorithms , 2008, 0803.1718.

[16]  Ron Meir,et al.  Lower bounds for multivariate approximation by affine-invariant dictionaries , 2001, IEEE Trans. Inf. Theory.

[17]  Holger Wendland,et al.  Optimal approximation orders in Lp for radial basis functions , 2000 .

[18]  M. Viana,et al.  Strange attractors in saddle-node cycles: prevalence and globality , 1996 .

[19]  Tianping Chen,et al.  Approximation capability to functions of several variables, nonlinear functionals and operators by radial basis function neural networks , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[20]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[21]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[22]  Vitaly Maiorov,et al.  On best approximation of classes by radial functions , 2003, J. Approx. Theory.

[23]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[24]  V. Temlyakov.  Approximation in Learning Theory , 2008 .

[25]  Martin Burger,et al.  Error Bounds for Approximation with Neural Networks , 2001, J. Approx. Theory.

[26]  Vitaly Maiorov,et al.  Pseudo-dimension and entropy of manifolds formed by affine-invariant dictionary , 2006, Adv. Comput. Math..

[27]  Robert Schaback,et al.  Approximation by radial basis functions with finitely many centers , 1996 .

[28]  Allan Pinkus,et al.  Lower bounds for approximation by MLP neural networks , 1999, Neurocomputing.

[29]  Martin D. Buhmann,et al.  Radial Basis Functions , 2021, Encyclopedia of Mathematical Geosciences.

[30]  Zongben Xu,et al.  Optimal rate of the regularized regression learning algorithm , 2011, Int. J. Comput. Math..

[31]  George G. Lorentz,et al.  Constructive Approximation , 1993, Grundlehren der mathematischen Wissenschaften.

[32]  Feilong Cao,et al.  The essential rate of approximation for radial function manifold , 2011 .

[33]  Kurt Jetter,et al.  Approximation with polynomial kernels and SVM classifiers , 2006, Adv. Comput. Math..

[34]  H. N. Mhaskar,et al.  Neural Networks for Optimal Approximation of Smooth and Analytic Functions , 1996, Neural Computation.

[35]  Vitaly Maiorov,et al.  On lower bounds in radial basis approximation , 2005, Adv. Comput. Math..

[36]  Holger Wendland,et al.  Scattered Data Approximation: Conditionally positive definite functions , 2004 .

[37]  D. Levin,et al.  On quasi-interpolation by radial basis functions with scattered centres , 1995 .

[38]  A. Bejancu.  The Uniform Convergence of Multivariate Natural Splines , 1997 .

[39]  Feilong Cao,et al.  The errors in simultaneous approximation by feed-forward neural networks , 2010, Neurocomputing.

[40]  Jooyoung Park,et al.  Approximation and Radial-Basis-Function Networks , 1993, Neural Computation.

[41]  F. Cao,et al.  The rate of approximation of Gaussian radial basis neural networks in continuous function space , 2013 .

[42]  Marcello Sanguineti,et al.  Dependence of Computational Models on Input Dimension: Tractability of Approximation and Optimization Tasks , 2012, IEEE Transactions on Information Theory.

[43]  Adam Krzyzak,et al.  A Distribution-Free Theory of Nonparametric Regression , 2002, Springer series in statistics.