Statistical basis of nonlinear hebbian learning and application to clustering

Abstract Recently, the extension of Hebbian learning to nonlinear units has received increased attention. Some successful applications of this learning rule to nonlinear principal component analysis have been reported as well, however, a fundamental understanding of the processing capability of this learning rule in the nonlinear setting is still lacking. In this paper, we pursue a better understanding of what the nonlinear unit is actually doing by exploring the statistical characteristics of the criterion function being optimized and interpreting the operation of the nonlinear activation as a probability integral transformation. To improve the computational capability of the nonlinear units, data preprocessing is suggested. This leads to the development of a two-layer network which consists of linear units in the first layer and nonlinear units in the second layer. The linear units capture and filter the linear aspect (low order correlations) of the data and the nonlinear units discover higher order correlations. Several potential applications are demonstrated through simulated data and previously analyzed data from real measurements. The relationship to exploratory data analysis in statistics is discussed.

[1]  Samuel Kotz,et al.  Continuous univariate distributions : distributions in statistics , 1970 .

[2]  Mohamad H. Hassoun,et al.  Nonlinear Hebbian rule: a statistical interpretation , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[3]  J. Karhunen,et al.  Nonlinear Hebbian Algorithms for Sinusoidal Frequency Estimation , 1992 .

[4]  I. S. Yenyukov.  Indences for projection pursuit , 1989 .

[5]  P. Hall.  On Polynomial-Based Projection Indices for Exploratory Projection Pursuit , 1989 .

[6]  Juha Karhunen,et al.  Learning of robust principal component subspace , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[7]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[8]  Lei Xu,et al.  Theories for unsupervised learning: PCA and its nonlinear extensions , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[9]  F. Palmieri,et al.  Hebbian learning and self-association in nonlinear neural networks , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[10]  Adam Prügel-Bennett,et al.  Unsupervised Hebbian Learning and the Shape of the Neuron Activation Function , 1993 .

[11]  J. Friedman.  Exploratory Projection Pursuit , 1987 .

[12]  C. Posse.  An effective two-dimensional projection pursuit algorithm , 1990 .

[13]  J. G. Taylor,et al.  ARTIFICIAL NEURAL NETWORKS, 2 , 1992 .

[14]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[15]  Vijay K. Rohatgi,et al.  Statistical Inference , 1984 .

[16]  Andrzej Cichocki,et al.  Robust estimation of principal components by using neural network learning algorithms , 1993 .

[17]  Daniel M. Kammen,et al.  Correlations in high dimensional or asymmetric data sets: Hebbian neuronal processing , 1991, Neural Networks.

[18]  Stephen Coombes,et al.  Learning higher order correlations , 1993, Neural Networks.

[19]  D. Freedman,et al.  Asymptotics of Graphical Projection Pursuit , 1984 .

[20]  J. Karhunen.  Optimization criteria and nonlinear PCA neural networks , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[21]  John W. Tukey,et al.  A Projection Pursuit Algorithm for Exploratory Data Analysis , 1974, IEEE Transactions on Computers.

[22]  Richard A. Johnson,et al.  Applied Multivariate Statistical Analysis , 1983 .

[23]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[24]  Alexander A. Lubischew.  On the Use of Discriminant Functions in Taxonomy , 1962 .

[25]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[26]  Adam Prügel-Bennett,et al.  Non-Linear Statistical Analysis and Self-Organizing Hebbian Networks , 1993, NIPS.

[27]  P. R. Fisk,et al.  Distributions in Statistics: Continuous Multivariate Distributions , 1971 .

[28]  Erkki Oja,et al.  Nonlinear PCA: Algorithms and Applications , 1993 .