Learning the dynamic nature of speech with back-propagation for sequences

Abstract A novel learning algorithm is proposed, called Back-Propagation for Sequences (BPS), for a particular class of dynamic neural networks in which some units have local feedback. These networks can be trained to respond to sequences of input patterns and seem particularly suited for phoneme recognition. They exhibit a forgetting behavior and consequently only recently past information is taken into account for classification purposes. BPS permits online weight updating and it has the same time complexity and space requirements as back-propagation (BP) applied to feedforward networks. We present experimental results for problems connected with Automatic Speech Recognition.

[1]  Lokendra Shastri,et al.  Speech recognition using connectionist networks , 1988 .

[2]  Yoshua Bengio,et al.  USE OF NEURAL NETWORKS FOR THE RECOGNITION OF PLACE , 1988 .

[3]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[4]  Barak A. Pearlmutter.  Learning State Space Trajectories in Recurrent Neural Networks , 1989, Neural Computation.

[5]  Hervé Bourlard,et al.  Speech dynamics and recurrent neural networks , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[6]  Giovanni Soda,et al.  Local Feedback Multilayered Networks , 1992, Neural Computation.

[7]  D Zipser,et al.  Learning the hidden structure of speech. , 1988, The Journal of the Acoustical Society of America.

[8]  Kurt Hornik,et al.  Neural networks and principal component analysis: Learning from examples without local minima , 1989, Neural Networks.

[9]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[10]  Alexander H. Waibel,et al.  Modular Construction of Time-Delay Neural Networks for Speech Recognition , 1989, Neural Computation.

[11]  Alberto Tesi,et al.  On the Problem of Local Minima in Backpropagation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[13]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[14]  Yoshua Bengio,et al.  Artificial neural networks and their application to sequence recognition , 1991 .

[15]  Yoshua Bengio,et al.  Programmable execution of multi-layered networks for automatic speech recognition , 1989, CACM.

[16]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[17]  Geoffrey E. Hinton,et al.  Learning sets of filters using back-propagation , 1987 .

[18]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[19]  Raymond L. Watrous.  Speech Recognition Using Connectionist Networks Dissertation Proposal , 1988 .

[20]  Michael I. Jordan.  Supervised learning and systems with excess degrees of freedom , 1988 .

[21]  Piero Cosi,et al.  Use of Multi-Layered Networks for Coding Speech with Phonetic Features , 1988, NIPS.

[22]  P. Frasconi,et al.  Local Feedback Multi-Layered Networks , 1992 .

[23]  Kiyohiro Shikano,et al.  Modularity and scaling in large phonemic neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[24]  M. Gori,et al.  BPS: a learning algorithm for capturing the dynamic nature of speech , 1989, International 1989 Joint Conference on Neural Networks.

[25]  Hervé Bourlard,et al.  Speech pattern discrimination and multilayer perceptrons , 1989 .