Fast Learning Algorithms for Feedforward Neural Networks

In order to improve the training speed of multilayer feedforward neural networks (MLFNN), we propose and explore two new fast backpropagation (BP) algorithms obtained: (1) by changing the error functions, in case using the exponent attenuation (or bell impulse) function and the Fourier kernel function as alternative functions; and (2) by introducing the hybrid conjugate-gradient algorithm of global optimization for dynamic learning rate to overcome the conventional BP learning problems of getting stuck into local minima or slow convergence. Our experimental results demonstrate the effectiveness of the modified error functions since the training speed is faster than that of existing fast methods. In addition, our hybrid algorithm has a higher recognition rate than the Polak-Ribieve conjugate gradient and conventional BP algorithms, and has less training time, less complication and stronger robustness than the Fletcher-Reeves conjugate-gradient and conventional BP algorithms for real speech data.

[1]  Emile Fiesler,et al.  High-order and multilayer perceptron initialization , 1997, IEEE Trans. Neural Networks.

[2]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[3]  Brijesh Verma,et al.  Fast training of multilayer perceptrons , 1997, IEEE Trans. Neural Networks.

[4]  Amir F. Atiya,et al.  An accelerated learning algorithm for multilayer perceptron networks , 1994, IEEE Trans. Neural Networks.

[5]  Anastasios N. Venetsanopoulos,et al.  Fast learning algorithms for neural networks , 1992 .

[6]  Ya-Xiang Yuan,et al.  An Efficient Hybrid Conjugate Gradient Method for Unconstrained Optimization , 2001, Ann. Oper. Res..

[7]  Jean Cea,et al.  Optimization - Theory and Algorithms , 1978 .

[8]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .

[9]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[10]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[11]  Farid U. Dowla,et al.  Backpropagation Learning for Multilayer Feed-Forward Neural Networks Using the Conjugate Gradient Method , 1991, Int. J. Neural Syst..

[12]  Mukul Agarwal,et al.  Three Methods to Speed up the Training of Feedforward and Feedback Perceptrons , 1997, Neural Networks.

[13]  Nicolaos B. Karayiannis,et al.  Accelerating the training of feedforward neural networks using generalized Hebbian rules for initializing the internal representations , 1996, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[14]  Duan Li,et al.  On Restart Procedures for the Conjugate Gradient Method , 2004, Numerical Algorithms.

[15]  Etienne Barnard,et al.  Avoiding false local minima by proper initialization of connections , 1992, IEEE Trans. Neural Networks.

[16]  Yu-Hong Dai,et al.  Some Properties of A New Conjugate Gradient Method , 1998 .

[17]  M. Powell.  Nonconvex minimization calculations and the conjugate gradient method , 1984 .

[18]  George D. Magoulas,et al.  Effective Backpropagation Training with Variable Stepsize , 1997, Neural Networks.

[19]  Shixin Cheng,et al.  Dynamic learning rate optimization of the backpropagation algorithm , 1995, IEEE Trans. Neural Networks.

[20]  Benedikt K. Humpert.  Improving back propagation with a new error function , 1994, Neural Networks.

[21]  Yutaka Fukuoka,et al.  A modified back-propagation method to avoid false local minima , 1998, Neural Networks.

[22]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[23]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[24]  Arjen van Ooyen,et al.  Improving the convergence of the back-propagation algorithm , 1992, Neural Networks.

[25]  Sang-Hoon Oh.  Improving the error backpropagation algorithm with a modified error function , 1997, IEEE Trans. Neural Networks.