Gestures and Lip Shape Integration for Cued Speech Recognition

In this article, automatic recognition of Cued Speech in French based on hidden Markov models (HMMs) is presented. Cued Speech is a visual mode, which uses hand shapes in different positions and in combination with lip-patterns of speech makes all the sounds of spoken language clearly understandable to deaf and hearing-impaired people. The aim of Cued Speech is to overcome the problems of lipreading and thus enable deaf children and adults to understand full spoken language. In this study, lip shape component is fused with hand component using also multistream HMM decision fusion to realize Cued Speech recognition, and continuous phoneme recognition experiments using data from a normal-hearing and a deaf cuer were conducted. In the case of the normal-hearing cuer, the obtained phoneme accuracy was 83.5%, and in the case of the deaf cuer 82.1%.

[1]  J. Leybaert,et al.  Phonology acquired through the eyes and spelling in deaf children. , 2000, Journal of experimental child psychology.

[2]  C M Reed,et al.  Automatic speech recognition to aid the hearing impaired: prospects for the automatic generation of cued speech. , 1994, Journal of rehabilitation research and development.

[3]  James Wc.  American association of mental deficiency presents panel on training the mentally retarded deaf. , 1967 .

[4]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Denis Beautemps,et al.  Cued speech recognition for augmentative communication in normal-hearing and hearing-impaired subjects , 2009, INTERSPEECH.

[6]  M. C. Jones.  Cued speech. , 1992, ASHA.

[7]  Michael G. Strintzis,et al.  Multimodal fusion for cued Speech language recognition , 2007, 2007 15th European Signal Processing Conference.

[8]  A. Montgomery,et al.  Physical characteristics of the lips underlying vowel lipreading performance. , 1983, The Journal of the Acoustical Society of America.

[9]  G. H. Nicholls,et al.  Cued Speech and the reception of spoken language. , 1982, Journal of speech and hearing research.

[10]  Hermann Ney,et al.  Speech recognition techniques for a sign language recognition system , 2007, INTERSPEECH.

[11]  Hervé Bourlard,et al.  A mew ASR approach based on independent processing and recombination of partial frequency bands , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[12]  Chalapathy Neti,et al.  Recent advances in the automatic recognition of audiovisual speech , 2003, Proc. IEEE.

[13]  Denis Beautemps,et al.  Lip Shape and Hand Position Fusion for Automatic Vowel Recognition in Cued Speech for French , 2009, IEEE Signal Processing Letters.