Image sequence analysis using a spatio-temporal coding for automatic lipreading

In this paper we present the details of a processing technique utilised to extract parameters from the images of a talking person's mouth region. These parameters are converted into impulse sequences for a given series of images. Spatio-temporal (ST) coding of the impulses provide an input to a spatio-temporal neural network architecture. The system is tested on a French digit recognition task and the results obtained are given. The overall procedure is sufficiently simple to allow us to make plans for its realisation as a real-time system.

[1]  Gilles Vaucher.  Neuro-Biological Bases for Spatio-Temporal Data Coding in Artificial Neural Networks , 1996, ICANN.

[2]  Georges Vaucher.  An algebra for recognition of spatio-temporal forms , 1997, ESANN.

[3]  Patrick van der Smagt,et al.  Introduction to neural networks , 1995, The Lancet.

[4]  G Vaucher.  An algebraic interpretation of PSP composition. , 1998, Bio Systems.

[5]  Peter E. Hart,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[6]  Q. Summerfield.  Some preliminaries to a comprehensive account of audio-visual speech perception. , 1987 .

[7]  Gilles Vaucher,et al.  A fully-neural solution for online handwritten character recognition , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[8]  Annick Le Glaunec,et al.  Human Faces Detection and Tracking in Video Sequence , 1995 .

[9]  Gregory J. Wolff,et al.  Neural network lipreading system for improved speech recognition , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[10]  Juergen Luettin,et al.  Visual Speech and Speaker Recognition , 1997 .

[11]  Abdul Rauf Baig,et al.  A spatio-temporal neural network applied to visual speech recognition , 1999 .

[12]  Alan Jeffrey Goldschen,et al.  Continuous automatic speech recognition by lipreading , 1993 .

[13]  Alexander H. Waibel,et al.  See Me, Hear Me: Integrating Automatic Speech Recognition and Lip-reading , 1994 .

[14]  Gilles Vaucher,et al.  A Spatio-temporal Perceptron for On-Line Handwritten Character Recognition , 1997, ICANN.