Analysis of Weight Initialization Routines for Scaled Conjugate Gradient Training Algorithm

The choice of weight initialization routines is one of the important choices to be made for improving the training efficiency of an artificial neural network. In this paper, we analyze the affect of many known weight initialization routines, on training of an artificial neural network, when it was trained with a second order scaled conjugate gradient training algorithm. A number of experiments were conducted to perform this analysis over eight selected function approximation problems. The results suggest that the partially deterministic weight initialization method and the Nguyen-Widrow initialization technique performed equally well and helped the network train and generalize better by achieving better training and simulation error values.

[1]  Martin Fodslette Møller,et al.  A scaled conjugate gradient algorithm for fast supervised learning , 1993, Neural Networks.

[2]  Pravin Chandra,et al.  A partially deterministic weight initialization method for SFFANNs , 2014, 2014 IEEE International Advance Computing Conference (IACC).

[3]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[4]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[5]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[6]  Kurt Hornik,et al.  Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks , 1990, Neural Networks.

[7]  Vladimir Cherkassky,et al.  Comparison of adaptive methods for function estimation from samples , 1996, IEEE Trans. Neural Networks.

[8]  Sandro Ridella,et al.  Statistically controlled activation weight initialization (SCAWI) , 1992, IEEE Trans. Neural Networks.

[9]  Vassilis P. Plagianakos,et al.  An Improved Backpropagation Method with Adaptive Learning Rate , 1998 .

[10]  Jong Beom Ra,et al.  Weight value initialization for improving training speed in the backpropagation network , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[11]  Pravin Chandra,et al.  Interval based Weight Initialization Method for Sigmoidal Feedforward Artificial Neural Networks , 2014 .

[12]  Emile Fiesler,et al.  High-order and multilayer perceptron initialization , 1997, IEEE Trans. Neural Networks.

[13]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[14]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[15]  C.-L. Chen,et al.  Improving the training speed of three-layer feedforward neural nets by optimal estimation of the initial weights , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.