A new learning method using prior information of neural networks

In this paper, we present a new learning method using prior information for three-layered neural networks. Usually when neural networks are used for identification of systems, all of their weights are trained independently, without considering their inter-relation of weight values. Thus the training results are not usually good. The reason for this is that each parameter has its influence on others during the learning. To overcome this problem, first, we give an exact mathematical equation that describes the relation between weight values given by a set of data conveying prior information. Then we present a new learning method that trains a part of the weights and calculates the others by using these exact mathematical equations. In almost all cases, this method keeps prior information given by a mathematical structure exactly during the learning. In addition, a learning method using prior information expressed by inequality is also presented. In any case, the degree of freedom of networks (the number of adjustable weights) is appropriately limited in order to speed up the learning and ensure small errors. Numerical computer simulation results are provided to support the present approaches.

[1]  Lars Kai Hansen,et al.  Regularization with a Pruning Prior , 1997, Neural Networks.

[2]  Jenq-Neng Hwang,et al.  Solving inverse problems by Bayesian neural network iterative inversion with ground truth incorporation , 1997, IEEE Trans. Signal Process..

[3]  Hyun Myung,et al.  Time-varying two-phase optimization and its application to neural-network learning , 1997, IEEE Trans. Neural Networks.

[4]  Jin Wang,et al.  Weight smoothing to improve network generalization , 1994, IEEE Trans. Neural Networks.

[5]  Michael A. Arbib,et al.  Generation of temporal sequences using local dynamic programming , 1995, Neural Networks.

[6]  Germano Lambert-Torres,et al.  A genetic-based neuro-fuzzy approach for modeling and control of dynamical systems , 1998, IEEE Trans. Neural Networks.

[7]  Juha Karhunen,et al.  Generalizations of principal component analysis, optimization problems, and neural networks , 1995, Neural Networks.

[8]  Abraham Kandel,et al.  Compensatory neurofuzzy systems with fast learning algorithms , 1998, IEEE Trans. Neural Networks.

[9]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[10]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[11]  Kazuyoshi Itoh,et al.  Generalization of shift invariant neural networks: Image processing of corneal endothelium , 1996, Neural Networks.

[12]  Stephen Grossberg,et al.  Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps , 1992, IEEE Trans. Neural Networks.

[13]  Peter M. Williams,et al.  Bayesian Regularization and Pruning Using a Laplace Prior , 1995, Neural Computation.

[14]  Sandro Ridella,et al.  Representation and generalization properties of class-entropy networks , 1999, IEEE Trans. Neural Networks.

[15]  Lipo Wang,et al.  Noise injection into inputs in sparsely connected Hopfield and winner-take-all neural networks , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[16]  Yoh-Han Pao,et al.  The ensemble approach to neural-network learning and generalization , 1999, IEEE Trans. Neural Networks.

[17]  Mehdi Mostaghimi,et al.  Bayesian estimation of a decision using information theory , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[18]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[19]  Tianping Chen,et al.  Approximation capability to functions of several variables, nonlinear functionals and operators by radial basis function neural networks , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[20]  Sang-Hoon Oh,et al.  다층 퍼셉트론 신경회로망의 학습가속을 위한 새로운 중간층 오차함수와 최적 학습률 = Fast training of multilayer perceptrons with a new error function for hidden layer and optimum learning rates , 1999 .