Neural Network Initialization

Proper initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. This publication aims at determining the optimal value of the initial weight variance (or range), which is the principal parameter of random weight initialization methods for both types of neural networks.

[1]  Sandro Ridella,et al.  Statistically controlled activation weight initialization (SCAWI) , 1992, IEEE Trans. Neural Networks.

[2]  John C. Platt.  Leaning by Combining Memorization and Gradient Descent , 1990, NIPS.

[3]  Thierry Denoeux,et al.  Initializing back propagation networks with prototypes , 1993, Neural Networks.

[4]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[5]  Emile Fiesler,et al.  Results on the Steepness in Backpropagation Neural Networks , 1994 .

[6]  G. Judge,et al.  The Theory and Practice of Econometrics , 1981 .

[7]  C.-L. Chen,et al.  Improving the training speed of three-layer feedforward neural nets by optimal estimation of the initial weights , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[8]  Etienne Barnard,et al.  Avoiding false local minima by proper initialization of connections , 1992, IEEE Trans. Neural Networks.

[9]  Emile Fiesler,et al.  Neural network classification and formalization , 1994 .

[10]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[11]  C. L. Giles,et al.  Machine learning using higher order correlation networks , 1986 .

[12]  Emile Fiesler,et al.  Modular Object-Oriented Neural Network Simulators and Topology Generalizations , 1994 .

[13]  John F. Kolen,et al.  Backpropagation is Sensitive to Initial Conditions , 1990, Complex Syst..

[14]  Jong Beom Ra,et al.  Weight value initialization for improving training speed in the backpropagation network , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[15]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[16]  Sang-Hoon Oh,et al.  An analysis of premature saturation in back propagation learning , 1993, Neural Networks.

[17]  Yoh-Han Pao,et al.  Adaptive pattern recognition and neural networks , 1989 .

[18]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .