Emergent componential coding of a handwritten image database by neural self-organisation.

This paper demonstrates the unsupervised discovery of localised components in real image data, using images of much larger size than the small fragments from which components have previously been extracted. The handwriting images used are also much more homogeneous than the random natural scenes used in earlier demonstrations, containing components of a specific size-scale and structure. Because of this homogeneity, the components found are not wavelets covering a range of size scales: instead, they correspond to line- and curve-segments made by the pen. The objective function that is optimised here encodes and reconstructs the data via a Markov process, and is also related to density modelling techniques. Several earlier theoretical and experimental results can also be attributed to the form of neuron used here, including the extraction of words from continuous speech and the discovery of unknown transformation invariances via the controlled breaking of dynamical symmetry.

[1]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[2]  H. B. Barlow,et al.  Unsupervised Learning , 1989, Neural Computation.

[3]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[4]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[5]  H. Sebastian Seung,et al.  Unsupervised Learning by Convex and Conic Coding , 1996, NIPS.

[6]  Geoffrey E. Hinton,et al.  Autoencoders, Minimum Description Length and Helmholtz Free Energy , 1993, NIPS.

[7]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[8]  R. Zemel.  A minimum description length framework for unsupervised learning , 1994 .

[9]  Chris J. S. Webber,et al.  Generalisation and discrimination emerge from a self-organising componential network: a speech example , 1997 .

[10]  Geoffrey E. Hinton,et al.  Varieties of Helmholtz Machine , 1996, Neural Networks.

[11]  C. Webber,et al.  Self-organisation of transformation-invariant detectors for constituents of perceptual patterns , 1994 .

[12]  C. Webber,et al.  Self-organization of position- and deformation-tolerant neural representations , 1991 .

[13]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[14]  Eric Saund,et al.  A Multiple Cause Mixture Model for Unsupervised Learning , 1995, Neural Computation.

[15]  John W. Tukey,et al.  A Projection Pursuit Algorithm for Exploratory Data Analysis , 1974, IEEE Transactions on Computers.

[16]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.