Nonlinear function approximation: Computing smooth solutions with an adaptive greedy algorithm

In contrast to linear schemes, nonlinear approximation techniques allow for dimension independent rates of convergence. Unfortunately, typical algorithms (such as, e.g., backpropagation) are not only computationally demanding, but also unstable in the presence of data noise. While we can show stability for a weak relaxed greedy algorithm, the resulting method has the drawback that it requires in practise unavailable smoothness information about the data. In this work we propose an adaptive greedy algorithm which does not need this information but rather recovers it iteratively from the available data. We show that the generated approximations are always at least as smooth as the original function and that the algorithm also remains stable, when it is applied to noisy data. Finally, the applicability of this algorithm is demonstrated by numerical experiments.

[1]  Ronald A. DeVore,et al.  Some remarks on greedy algorithms , 1996, Adv. Comput. Math..

[2]  L. Jones.  A Simple Lemma on Greedy Approximation in Hilbert Space and Convergence Rates for Projection Pursuit Regression and Neural Network Training , 1992 .

[3]  Vladimir N. Temlyakov,et al.  Weak greedy algorithms[*]This research was supported by National Science Foundation Grant DMS 9970326 and by ONR Grant N00014‐96‐1‐1003. , 2000, Adv. Comput. Math..

[4]  R. DeVore,et al.  Optimal nonlinear approximation , 1989 .

[5]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[6]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[7]  Irwin W. Sandberg,et al.  A note on error bounds for approximation in inner product spaces , 1996 .

[8]  Camillo Melzi.  Computing potentials on a periodic bidimensional grid , 1996 .

[9]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[10]  Federico Girosi,et al.  Generalization bounds for function approximation from scattered noisy data , 1999, Adv. Comput. Math..

[11]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[12]  Martin Burger,et al.  Regularized Greedy Algorithms for Network Training with Data Noise , 2005, Computing.

[13]  L. Jones.  On a conjecture of Huber concerning the convergence of projection pursuit regression , 1987 .

[14]  Martin Burger,et al.  Regularized data-driven construction of fuzzy controllers , 2002 .

[15]  L. Jones.  Local greedy approximation for nonlinear regression and neural network training , 2000 .

[16]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[17]  C. Darken,et al.  Constructive Approximation Rates of Convex Approximation in Non-hilbert Spaces , 2022 .

[18]  Ward Cheney,et al.  A course in approximation theory , 1999 .

[19]  A. Pinkus.  n-Widths in Approximation Theory , 1985 .

[20]  Martin Burger,et al.  Error Bounds for Approximation with Neural Networks , 2001, J. Approx. Theory.

[21]  Peter L. Bartlett,et al.  Efficient agnostic learning of neural networks with bounded fan-in , 1996, IEEE Trans. Inf. Theory.

[22]  G. Walter,et al.  Remarks on projection pursuit regression and density estimation , 1992 .

[23]  Vladimir N. Temlyakov,et al.  Nonlinear Methods of Approximation , 2003, Found. Comput. Math..

[24]  S. Smale,et al.  Shannon sampling and function reconstruction from point values , 2004 .