An Improved Conjugate Gradient Based Learning Algorithm for Back Propagation Neural Networks

The conjugate gradient optimization algorithm is combined with the modified back propagation algorithm to yield a computationally efficient algorithm for training multilayer perceptron (MLP) networks (CGFR/AG). The computational efficiency is enhanced by adaptively modifying initial search direction as described in the following steps: (1) Modification on standard back propagation algorithm by introducing a gain variation term in the activation function, (2) Calculation of the gradient descent of error with respect to the weights and gains values and (3) the determination of a new search direction by using information calculated in step (2). The performance of the proposed method is demonstrated by comparing accuracy and computation time with the conjugate gradient algorithm used in MATLAB neural network toolbox. The results show that the computational efficiency of the proposed method was better than the standard conjugate gradient algorithm. Keywords—Adaptive gain variation, back-propagation, activation function, conjugate gradient, search direction.

[1]  Adrian J. Shepherd,et al.  Second-order methods for neural networks - fast and reliable training methods for multi-layer perceptrons , 1997, Perspectives in neural computing.

[2]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[3]  H. Y. Huang.  Unified approach to quadratically convergent algorithms for function minimization , 1970 .

[4]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[5]  Emile Fiesler,et al.  The Interchangeability of Learning Rate and Gain in Backpropagation Neural Networks , 1996, Neural Computation.

[6]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[7]  Holger R. Maier,et al.  The effect of internal parameters and geometry on the performance of back-propagation neural networks: an empirical study , 1998 .

[8]  D. E. Rumelhart,et al.  Learning internal representations by back-propagating errors , 1986 .

[9]  C. H. Chen,et al.  An Empirical Study Of The Gradient Descent And The Conjugate Gradient Backpropagation Neural Networks , 1992, OCEANS 92 Proceedings@m_Mastering the Oceans Through Technology.

[10]  Guo-An Chen,et al.  Acceleration of backpropagation learning using optimised learning rate and momentum , 1993 .

[11]  Michael K. Weir,et al.  A method for self-determination of adaptive learning rates in back propagation , 1991, Neural Networks.

[12]  Z. Zainuddin,et al.  Improving the Convergence of the Backpropagation Algorithm Using Local Adaptive Techniques , 2007, International Conference on Computational Intelligence.

[13]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[14]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[15]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[16]  Yogesh Singh,et al.  An activation function adapting training algorithm for sigmoidal feedforward networks , 2004, Neurocomputing.

[17]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..