Neural Network Learning as an Inverse Problem

Capability of generalization in learning of neural networks from examples can be modelled using regularization, which has been developed as a tool for improving stability of solutions of inverse problems. Such problems are typically described by integral operators. It is shown that learning from examples can be reformulated as an inverse problem defined by an evaluation operator. This reformulation leads to an analytical description of an optimal input/output function of a network with kernel units, which can be employed to design a learning algorithm based on a numerical solution of a system of linear equations.

[1]  E. Parzen.  An Approach to Time Series Analysis , 1961 .

[2]  L. Galway.  Spline Models for Observational Data , 1991 .

[3]  M. Bertero.  Linear Inverse and III-Posed Problems , 1989 .

[4]  M. Aizerman,et al.  Theoretical Foundations of the Potential Function Method in Pattern Recognition Learning , 1964 .

[5]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[6]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[7]  Marcello Sanguineti,et al.  Error Estimates for Approximate Optimization by the Extended Ritz Method , 2005, SIAM J. Optim..

[8]  Per Christian Hansen,et al.  Rank-Deficient and Discrete Ill-Posed Problems , 1996 .

[9]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[10]  Marcello Sanguineti,et al.  Learning with generalization capability by kernel methods of bounded complexity , 2005, J. Complex..

[11]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[12]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[13]  Paul J. Werbos.  Backpropagation: basics and new developments , 1998 .

[14]  Federico Girosi,et al.  An Equivalence Between Sparse Approximation and Support Vector Machines , 1998, Neural Computation.

[15]  Mustafa.  Abstract , 1952 .

[16]  Åke Björck,et al.  Numerical methods for least square problems , 1996 .

[17]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[18]  C. W. Groetsch,et al.  Generalized inverses of linear operators , 1977 .

[19]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[20]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[21]  T. Poggio,et al.  The Mathematics of Learning: Dealing with Data , 2005, 2005 International Conference on Neural Networks and Brain.

[22]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[23]  Dustin Boswell,et al.  Introduction to Support Vector Machines , 2002 .

[24]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[25]  Vladimir Cherkassky,et al.  The Nature Of Statistical Learning Theory , 1997, IEEE Trans. Neural Networks.