Probabilistic design of layered neural networks based on their unified framework

Proposes three ways of designing artificial neural networks based on a unified framework and uses them to develop new models. First, the authors show that artificial neural networks can be understood as probability density functions with parameters. Second, the authors propose three design methods for new models: a method for estimating the occurrence probability of the inputs, a method for estimating the variance of the outputs, and a method for estimating the simultaneous probability of inputs and outputs. Third, the authors design three new models using the proposed methods: a neural network with occurrence probability estimation, a neural network with output variance estimation, and a probability competition neural network. The authors' experimental results show that the proposed neural networks have important abilities in information processing; they can tell how often a given input occurs, how widely the outputs are distributed, and from what kinds of inputs a given output is inferred.

[1]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[2]  H. Akaike.  A new look at the statistical model identification , 1974 .

[3]  L. Gleser,et al.  Classical asymptotic properties of a certain estimator related to the maximum likelihood estimator , 1975 .

[4]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[5]  Jorma Rissanen,et al.  Universal coding, information, prediction, and estimation , 1984, IEEE Trans. Inf. Theory.

[6]  Esther Levin,et al.  A statistical approach to learning and generalization in layered neural networks , 1989, Proc. IEEE.

[7]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[8]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[9]  Halbert White,et al.  Learning in Artificial Neural Networks: A Statistical Perspective , 1989, Neural Computation.

[10]  Donald F. Specht,et al.  Probabilistic neural networks , 1990, Neural Networks.

[11]  Donald F. Specht,et al.  A general regression neural network , 1991, IEEE Trans. Neural Networks.

[12]  Bart Kosko.  Stochastic competitive learning , 1991, IEEE Trans. Neural Networks.

[13]  I. Grabec,et al.  Automatic modeling of physical phenomena : application to ultrasonic data , 1991 .

[14]  S. Watanabe,et al.  An ultrasonic 3D object recognition method based on the unified neural network theory , 1992, IEEE 1992 Ultrasonics Symposium Proceedings.

[15]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[16]  Shun-ichi Amari,et al.  Four Types of Learning Curves , 1992, Neural Computation.

[17]  Sumio Watanabe.  An Optimization Method of Layered Neural Networks based on the Modified Information Criterion , 1993, NIPS.

[18]  Shun-ichi Amari,et al.  Network information criterion-determining the number of hidden units for an artificial neural network model , 1994, IEEE Trans. Neural Networks.