A homotopy method for training neural networks

Abstract In many fields of signal processing feed-forward neural networks, especially multilayer perceptron neural networks (MLP-NNs), are used as approximators. On the grounds of expedience the parameter (weight) adaptation process (training) is formulated as an optimization procedure solving a conventional nonlinear regression problem, which is a very important practical task. Thus, the presented theory can easily be adapted to any similar problem. This paper presents an innovative approach to minimize the training error. After a theoretic foundation we will demonstrate that employing the homotopy method (HM) in a second-order optimization technique leads to much better convergence properties than direct methods. Simulation results on various examples illustrate excellent robustness concerning the initial values of the weights and less overall computational costs. Even though this paper addresses learning of neural networks it is outlined in general manner as far as possible motivating to apply the homotopy method to any related parameter optimization problem in signal processing.

[1]  Wolfram Schiffmann,et al.  Application of Genetic Algorithms to the Construction of Topologies for Multilayer Perceptrons , 1993 .

[2]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[3]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[4]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[5]  E. Allgower,et al.  Numerical Continuation Methods , 1990 .

[6]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[7]  A. Owens,et al.  Efficient training of the backpropagation network by solving a system of stiff ordinary differential equations , 1989, International 1989 Joint Conference on Neural Networks.

[8]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[9]  Wolfram Schiffmann,et al.  Optimization of the Backpropagation Algorithm for Training Multilayer Perceptrons , 1994 .

[10]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[11]  Peter E. Hart,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[12]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[13]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[14]  Simon Haykin,et al.  Neural networks , 1994 .

[15]  N. D. Villiers,et al.  A Continuation Method for Nonlinear Regression , 1981 .

[16]  K. Venkatesh Prasad.  BOOK REVIEW: "Neural Networks for Optimization and Signal Processing", A. Cichocki and R. Unbehauen , 1993, Int. J. Neural Syst..

[17]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[18]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[19]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[20]  J. Loewenthal.  DECISION , 1969, Definitions.

[21]  A. Dale Magoun,et al.  Decision, estimation and classification , 1989 .

[22]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[23]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.