On complexity analysis of supervised MLP-learning for algorithmic comparisons

This paper presents the complexity analysis of a standard supervised MLP-learning algorithm in conjunction with the well-known backpropagation, an efficient method for evaluation of derivatives, in either batch or incremental learning mode. In particular, we detail the cost per epoch (i.e., operations required for processing one sweep of all the training data) using "approximate" FLOPs (floating point operations) in a typical backpropagation for solving neural networks nonlinear least squares problems. Furthermore, we identify erroneous complexity analyses found in the past NN literature. Our operation-count formula would be very useful for a given MLP architecture to compare learning algorithms.

[1]  Stuart E. Dreyfus,et al.  On derivation of MLP backpropagation from the Kelley-Bryson optimal-control gradient formula and its application , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[2]  Peter Tang,et al.  The Computation of Transcendental Functions on the IA-64 Architecture , 1999 .

[3]  Adrian J. Shepherd,et al.  Second-order methods for neural networks - fast and reliable training methods for multi-layer perceptrons , 1997, Perspectives in neural computing.

[4]  Toshinobu Yoshida.  Rapid learning method for multilayered neural networks using two-dimensional conjugate gradient search , 1992 .

[5]  Martin Fodslette Møller,et al.  A scaled conjugate gradient algorithm for fast supervised learning , 1993, Neural Networks.

[6]  Adrian J. Shepherd,et al.  Second-Order Methods for Neural Networks , 1997 .

[7]  James Demmel,et al.  On Iterative Krylov-Dogleg Trust-Region Steps for Solving Neural Networks Nonlinear Least Squares Problems , 2000, NIPS.

[8]  Martin Bouchard,et al.  New recursive-least-squares algorithms for nonlinear active control of sound and vibration using neural networks , 2001, IEEE Trans. Neural Networks.

[9]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[10]  George D. Magoulas,et al.  Improving the Convergence of the Backpropagation Algorithm Using Learning Rate Adaptation Methods , 1999, Neural Computation.