Statistical mechanics of unsupervised structure recognition

A model of unsupervised learning is studied, where the environment provides N-dimensional input examples that are drawn from two overlapping Gaussian clouds. We consider the optimization of two different objective functions: the search for the direction of the largest variance in the data and the largest separating gap (stability) between clusters of examples respectively. By means of a statistical-mechanics analysis, we investigate how well the underlying structure is inferred from a set of examples. The performances of the learning algorithms depend crucially on the actual shape of the input distribution. A generic result is the existence of a critical number of examples needed for successful learning. The learning strategies are compared with methods different in spirit, such as the estimation of parameters in a model distribution and an information-theoretical approach.

[1]  Y. Chien,et al.  Pattern classification and scene analysis , 1974 .

[2]  Peter E. Hart,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[3]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[4]  W. Krauth,et al.  Learning algorithms with optimal stability in neural networks , 1987 .

[5]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[6]  P. Leath,et al.  The failure distribution in percolation models of breakdown , 1987 .

[7]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[8]  E. Gardner,et al.  Optimal storage properties of neural network models , 1988 .

[9]  E. Gardner.  The space of interactions in neural network models , 1988 .

[10]  Michael Biehl,et al.  The AdaTron: An Adaptive Perceptron Algorithm , 1989 .

[11]  M. Opper,et al.  On the ability of the optimal perceptron to generalise , 1990 .

[12]  G. Hartmann,et al.  Parallel Processing in Neural Systems and Computers , 1990 .

[13]  Suzanna Becker,et al.  Unsupervised Learning Procedures for Neural Networks , 1991, Int. J. Neural Syst..

[14]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[15]  H. Gutfreund,et al.  Learning and retrieval in attractor neural networks above saturation , 1991 .

[16]  Sompolinsky,et al.  Statistical mechanics of learning from examples. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[17]  M. Benaim,et al.  A stochastic model of neural network for unsupervised learning , 1992 .

[18]  P. Rujan.  A Fast Method for Calculating the Perceptron with Maximal Stability , 1993 .

[19]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[20]  Adam Prügel-Bennett,et al.  Statistical mechanics of unsupervised Hebbian learning , 1993 .

[21]  Michael Biehl,et al.  Statistical Mechanics of Unsupervised Learning , 1993 .

[22]  Sompolinsky,et al.  Scaling laws in learning of classification tasks. , 1993, Physical review letters.

[23]  Néstor Parga,et al.  Duality Between Learning Machines: A Bridge Between Supervised and Unsupervised Learning , 1994, Neural Computation.

[24]  J. Nadal,et al.  Optimal unsupervised learning , 1994 .