MDS: a multimodal-based dialog system

This paper describes MDS: a Multimodal-based Dialog System that supports communication between the hearing impaired and hearing-abled. The system converts sign language to speech, and combines speech with gesture and lip motion using a human face. The features of the human face are derived by doing a 3D feature extraction of the speaker's face, so that the “virtual face” similar to the actual speaker. The main technologies associated with the system include sign language recognition, sign language synthesis and the synchrony of the lip movement and speech. Integration of the sign language recognition, sign language synthesis, speech recognition, speech synthesis and 3D virtual human technologies provides a new way to interact with computers for the hearing impaired.

[1]  Thomas S. Huang,et al.  Human face detection in a complex background , 1994, Pattern Recognit..

[2]  Ho-Sub Yoon,et al.  Recognition of alphabetical hand gestures using hidden Markov model , 1999 .

[3]  Wen Gao,et al.  A continuous Chinese sign language recognition system , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[4]  Dimitris N. Metaxas,et al.  Parallel hidden Markov models for American sign language recognition , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.