Variational Approach to Unsupervised Learning Algorithms of Neural Networks

Variational approach in pattern recognition basing on a mean risk functional is applied to unsupervised learning algorithms for neural networks. The paper summarizes main statements and results of the approach: a structure of mean risk functionals relevant for the problem of grouping the multivariate data; optimum conditions; recurrent algorithms of stochastic approximation type for minimization of the functionals; and conditions ensuring convergence of the algorithms to appropriate solutions. Basing on this theory a general scheme is proposed and it allows to obtain neural networks whose adaptation laws arise as the recurrent unsupervised learning algorithms for suitably selected mean risk functionals. The well known Grossberg networks, counter-propagation networks, and Kohonen self-organizing networks are among them. Moreover the new variants of unsupervised-learning neural structures can be derived as it is demonstrated in the paper. The developed mathematical means can serve for theoretical justification of using such neural networks in data processing systems. Copyright 1997 Elsevier Science Ltd. All Rights Reserved.

[1]  M. T. Wasan.  Stochastic Approximation , 1969 .

[2]  E. Oja,et al.  An Analysis of Convergence for a Learning Version of the Subspace Method , 1983 .

[3]  Gail A. Carpenter,et al.  Neural network models for pattern recognition and associative memory , 1989, Neural Networks.

[4]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[5]  Behrooz Kamgar-Parsi,et al.  Simultaneous fitting of several planes to point sets using neural networks , 1990, Comput. Vis. Graph. Image Process..

[6]  GrossbergS..  Adaptive pattern classification and universal recoding , 1976 .

[7]  T. Kohonen.  Self-organized formation of topographically correct feature maps , 1982 .

[8]  Laura Ignizio Burke,et al.  Clustering characterization of adaptive resonance , 1991, Neural Networks.

[9]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[10]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[11]  Erick R. Malaret,et al.  Classification performance of a Hopfield neural network based on a Hebbian-like learning rule , 1989, IEEE Trans. Inf. Theory.

[12]  Stephen Grossberg,et al.  Art 2: Self-Organization Of Stable Category Recognition Codes For Analog Input Patterns , 1988, Other Conferences.

[13]  R. Hecht-Nielsen.  Counterpropagation networks. , 1987, Applied optics.

[14]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[15]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[16]  Klaus Pawelzik,et al.  Quantifying the neighborhood preservation of self-organizing feature maps , 1992, IEEE Trans. Neural Networks.

[17]  Kumpati S. Narendra,et al.  Adaptation and learning in automatic systems , 1974 .

[18]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[19]  K. Urahama.  Global stability of some classes of neural networks , 1989 .

[20]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[21]  Svante Wold,et al.  Pattern recognition by means of disjoint principal components models , 1976, Pattern Recognit..

[22]  Eric Goles Ch.,et al.  Energy Functions in Neural Networks with Continuous Local Functions , 1989, Complex Systems.

[23]  B. S. Duran,et al.  Cluster Analysis: A Survey , 1974 .

[24]  Brian Everitt,et al.  Cluster analysis , 1974 .

[25]  David M. Clark,et al.  A convergence theorem for Grossberg learning , 1990, Neural Networks.

[26]  Frederick R. Fromm,et al.  Class: A nonparametric clustering algorithm , 1976, Pattern Recognit..

[27]  R.P. Lippmann,et al.  Pattern classification using neural networks , 1989, IEEE Communications Magazine.