A neural network training algorithm utilizing multiple sets of linear equations

A fast algorithm is presented for the training of multilayer perceptron neural networks. In each iteration, there are two passes through the training data. In the first pass, linear equations are solved for the output weights. In the second data pass, linear equations are solved for hidden unit weight changes. Full batching is used in both data passes. An algorithm is described for calculating the learning factor for use with the hidden weights. It is shown that the technique is significantly faster than standard output weight optimization-backpropagation.

[1]  Michael T. Manry,et al.  Surface parameter retrieval using fast learning neural networks , 1993 .

[2]  Timothy Masters,et al.  Neural, Novel & Hybrid Algorithms for Time Series Prediction , 1995 .

[3]  Panos J. Antsaklis,et al.  A simple method to derive bounds on the size and to train multilayer neural networks , 1991, IEEE Trans. Neural Networks.

[4]  James A. Anderson,et al.  An Introduction To Neural Networks , 1998 .

[5]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[6]  William H. Press,et al.  Numerical recipes , 1990 .

[7]  Nazif Tepedelenlioglu,et al.  A fast new algorithm for training feedforward neural networks , 1992, IEEE Trans. Signal Process..

[8]  Tommy W. S. Chow,et al.  Extended least squares based algorithm for training feedforward networks , 1997, IEEE Trans. Neural Networks.

[9]  Michael T. Manry,et al.  Backpropagation representation theorem using power series , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[10]  Adrian K. Fung,et al.  Backscattering from a randomly rough dielectric surface , 1992, IEEE Trans. Geosci. Remote. Sens..

[11]  Kevin N. Gurney,et al.  An introduction to neural networks , 2018 .

[12]  Michael T. Manry,et al.  Inversion of Surface Parameters Using Fast Learning Neural Networks , 1992, [Proceedings] IGARSS '92 International Geoscience and Remote Sensing Symposium.

[13]  Yousen Xia.  Neural network for solving extended linear programming problems , 1997, IEEE Trans. Neural Networks.

[14]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[15]  Stefanos Kollias,et al.  An adaptive least squares algorithm for the efficient training of artificial neural networks , 1989 .

[16]  Michael T. Manry,et al.  Conventional modeling of the multilayer perceptron using polynomial basis functions , 1993, IEEE Trans. Neural Networks.

[17]  Mu-Song Chen,et al.  Constructive proof of efficient pattern storage in the multi-layer perceptron , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[18]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[19]  M.-S. Chen,et al.  Nonlinear modelling of back-propagation neural networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[20]  George W. Irwin,et al.  A hybrid linear/nonlinear training algorithm for feedforward neural networks , 1998, IEEE Trans. Neural Networks.

[21]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[22]  Martin T. Hagan,et al.  Levenberg-Marquardt training for modular networks , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[23]  Simon A. Barton,et al.  A Matrix Method for Optimizing a Neural Network , 1991, Neural Computation.

[24]  W. Murray.  Numerical Methods for Unconstrained Optimization , 1975 .

[25]  P. J. Werbos,et al.  Backpropagation: past and future , 1988, IEEE 1988 International Conference on Neural Networks.

[26]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[27]  Philip E. Gill,et al.  Practical optimization , 1981 .

[28]  Michael T. Manry,et al.  Neural subnet design by direct polynomial mapping , 1992, IEEE Trans. Neural Networks.

[29]  Alan F. Murray,et al.  IEEE International Conference on Neural Networks , 1997 .

[30]  Michael T. Manry,et al.  Fast training of neural networks for remote sensing , 1994 .

[31]  Kurt Hornik,et al.  Neural networks and principal component analysis: Learning from examples without local minima , 1989, Neural Networks.

[32]  Yoh-Han Pao,et al.  Adaptive pattern recognition and neural networks , 1989 .

[33]  A. Fung.  Microwave Scattering and Emission Models and their Applications , 1994 .