Generalisation and discrimination emerge from a self-organising componential network: a speech example

It is demonstrated that a componential code emerges when a self-organising neural network is exposed to continuous speech. The code's components correspond to substructures that occur relatively independently of one another: words and phones. A capability for generalisation and discrimination develops without having been optimised explicitly. The componential structure is revealed by optimising a necessarily complicated nonlinear moment of the data's distribution, equal to the mean-squared output response of a multi-layered network of simple threshold neurons. Earlier analytical work had predicted that componential codes, generalisation and discrimination should emerge from the self-organisation of threshold neurons of this form, assuming certain properties of the pattern-space distribution of the data.

[1]  E. McDermott,et al.  A hybrid speech recognition system using HMMs with an LVQ-trained codebook , 1990 .

[2]  John S. Bridle,et al.  Alpha-nets: A recurrent 'neural' network architecture with a hidden Markov model interpretation , 1990, Speech Commun..

[3]  Abraham Lempel,et al.  A universal algorithm for sequential data compression , 1977, IEEE Trans. Inf. Theory.

[4]  C. Webber,et al.  Self-organisation of transformation-invariant detectors for constituents of perceptual patterns , 1994 .

[5]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[6]  Olli Ventä,et al.  Phonetic typewriter for Finnish and Japanese , 1988, ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing.

[7]  Roger K. Moore,et al.  RSRE (Royal Signals and Radar Establishment) Speech Database Recordings (1983). Part 2. Recording Made for Automatics Speech Recognition Assessment and Research , 1984 .

[8]  Geoffrey E. Hinton,et al.  Varieties of Helmholtz Machine , 1996, Neural Networks.

[9]  Ah Chung Tsoi,et al.  Locally recurrent globally feedforward networks: a critical review of architectures , 1994, IEEE Trans. Neural Networks.

[10]  R. Zemel.  A minimum description length framework for unsupervised learning , 1994 .

[11]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[12]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[13]  Roger K. Moore,et al.  The application of dynamic programming techniques to non-word based topic spotting , 1995, EUROSPEECH.

[14]  Stephen Grossberg,et al.  The ART of adaptive pattern recognition by a self-organizing neural network , 1988, Computer.

[15]  S. P. Luttrell,et al.  Self-supervised adaptive networks , 1992 .

[16]  Kunihiko Fukushima,et al.  Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position , 1982, Pattern Recognit..

[17]  John Makhoul,et al.  Discriminant analysis and supervised vector quantization for continuous speech recognition , 1990, International Conference on Acoustics, Speech, and Signal Processing.

[18]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[19]  Anthony J. Robinson,et al.  An application of recurrent nets to phone probability estimation , 1994, IEEE Trans. Neural Networks.

[20]  David Clarke,et al.  CGPC with guaranteed stability properties , 1992 .

[21]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[22]  Nathan Intrator,et al.  Feature Extraction Using an Unsupervised Neural Network , 1992, Neural Computation.

[23]  Shigeru Katagiri,et al.  Shift-invariant, multi-category phoneme recognition using Kohonen's LVQ2 , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[24]  Eric Saund,et al.  A Multiple Cause Mixture Model for Unsupervised Learning , 1995, Neural Computation.

[25]  Geoffrey E. Hinton,et al.  Autoencoders, Minimum Description Length and Helmholtz Free Energy , 1993, NIPS.

[26]  John W. Tukey,et al.  A Projection Pursuit Algorithm for Exploratory Data Analysis , 1974, IEEE Transactions on Computers.