Regularized Greedy Algorithms for Network Training with Data Noise

The aim of this paper is to construct a modified greedy algorithm applicable for an ill-posed function approximation problem in presence of data noise. We provide a detailed convergence analysis of the algorithm in presence of noise, and discuss the choice of the iteration parameters. This yields a stopping rule for which the corresponding algorithm is a regularization method with convergence rates in L2 and under weak additional assumptions also in Sobolev-spaces of positive order.Finally, we discuss the application of the modified greedy algorithm to sigmoidal neural networks and radial basis functions, and supplement the theoretical results by numerical experiments.

[1]  Martin Burger,et al.  Training neural networks with noisy data as an ill-posed problem , 2000, Adv. Comput. Math..

[2]  M. Burger,et al.  Regularized Greedy Algorithms for Neural Network Training with Data Noise , 2003 .

[3]  Irwin W. Sandberg,et al.  A note on error bounds for approximation in inner product spaces , 1996 .

[4]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[5]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[6]  Werner Linde,et al.  Infinitely divisible and stable measures on Banach spaces , 1983 .

[7]  Martin Burger,et al.  Regularized data-driven construction of fuzzy controllers , 2002 .

[8]  J. Lions,et al.  Non-homogeneous boundary value problems and applications , 1972 .

[9]  L. Jones.  A Simple Lemma on Greedy Approximation in Hilbert Space and Convergence Rates for Projection Pursuit Regression and Neural Network Training , 1992 .

[10]  Martin Burger,et al.  Analysis of Tikhonov regularization for function approximation by neural networks , 2003, Neural Networks.

[11]  Jacques-Louis Lions,et al.  Hilbert Theory of Trace and Interpolation Spaces , 1972 .

[12]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[13]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[14]  Camillo Melzi.  Computing potentials on a periodic bidimensional grid , 1996 .

[15]  H. Engl,et al.  Regularization of Inverse Problems , 1996 .