Magnified gradient function with deterministic weight modification in adaptive learning

This work presents two novel approaches, backpropagation (BP) with magnified gradient function (MGFPROP) and deterministic weight modification (DWM), to speed up the convergence rate and improve the global convergence capability of the standard BP learning algorithm. The purpose of MGFPROP is to increase the convergence rate by magnifying the gradient function of the activation function, while the main objective of DWM is to reduce the system error by changing the weights of a multilayered feedforward neural network in a deterministic way. Simulation results show that the performance of the above two approaches is better than BP and other modified BP algorithms for a number of learning problems. Moreover, the integration of the above two approaches forming a new algorithm called MDPROP, can further improve the performance of MGFPROP and DWM. From our simulation results, the MDPROP algorithm always outperforms BP and other modified BP algorithms in terms of convergence rate and global convergence capability.

[1]  Arjen van Ooyen,et al.  Improving the convergence of the back-propagation algorithm , 1992, Neural Networks.

[2]  I ScottKirkpatrick.  Optimization by Simulated Annealing: Quantitative Studies , 1984 .

[3]  Guo-An Chen,et al.  Acceleration of backpropagation learning using optimised learning rate and momentum , 1993 .

[4]  George D. Magoulas,et al.  Improving the Convergence of the Backpropagation Algorithm Using Learning Rate Adaptation Methods , 1999, Neural Computation.

[5]  Kurt Hornik,et al.  Convergence of learning algorithms with constant learning rates , 1991, IEEE Trans. Neural Networks.

[6]  Alberto Tesi,et al.  On the Problem of Local Minima in Backpropagation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[8]  Scott Kirkpatrick,et al.  Optimization by simulated annealing: Quantitative studies , 1984 .

[9]  Sang-Hoon Oh,et al.  An analysis of premature saturation in back propagation learning , 1993, Neural Networks.

[10]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[11]  M. O. Tokhi,et al.  Training neural networks: backpropagation vs. genetic algorithms , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[12]  Brijesh Verma,et al.  Fast training of multilayer perceptrons , 1997, IEEE Trans. Neural Networks.

[13]  Yonathan Bard,et al.  Nonlinear parameter estimation , 1974 .

[14]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[15]  Tamás D. Gedeon,et al.  Simulated annealing and weight decay in adaptive learning: the SARPROP algorithm , 1998, IEEE Trans. Neural Networks.

[16]  Mukul Agarwal,et al.  Three Methods to Speed up the Training of Feedforward and Feedback Perceptrons , 1997, Neural Networks.

[17]  Edward K. Blum,et al.  Approximation theory and feedforward networks , 1991, Neural Networks.

[18]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[19]  T. Brubaker,et al.  Nonlinear Parameter Estimation , 1979 .

[20]  D. J. Newman,et al.  UCI Repository of Machine Learning Database , 1998 .

[21]  Jaques Reifman,et al.  Premature Saturation in Backpropagation Networks: Mechanism and Necessary Conditions , 1995, Neural Networks.

[22]  Adi Ben-Israel,et al.  Generalized inverses: theory and applications , 1974 .

[23]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[24]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[25]  Bin-Da Liu,et al.  A backpropagation algorithm with adaptive learning rate and momentum coefficient , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[26]  Scott Kirkpatrick,et al.  Optimization by Simmulated Annealing , 1983, Sci..