A MULTIMODAL FRAMEWORK FOR THE COMMUNICATION OF THE DISABLED

In this paper, a novel system, which aims to provide alternative tools and interfaces to blind and deaf-and-mute people and enable their communication and interaction with the computer is presented. All the involved technologies are integrated into a treasure hunting game application that is jointly played by the blind and deaf-and-mute user. The integration of the multimodal interfaces into a game application serves both as an entertainment and a pleasant education tool to the users. The proposed application integrates haptics, audio, visual output and computer vision, sign language analysis and synthesis, speech recognition and synthesis, in order to provide an interactive environment where the blind and deaf-and-mute users can collaborate to play the treasure hunting game.

[1]  A. Murat Tekalp,et al.  Multimodal speaker identification using an adaptive classifier cascade based on modality reliability , 2005, IEEE Transactions on Multimedia.

[2]  Min C. Shin,et al.  Effect of colorspace transformation, the illuminance component, and color modeling on skin detection , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[3]  Stephen A. Brewster,et al.  A paradigm shift: alternative interaction techniques for use with mobile & wearable devices , 2003, CASCON.

[4]  Kevin P. Murphy,et al.  Dynamic Bayesian Networks for Audio-Visual Speech Recognition , 2002, EURASIP J. Adv. Signal Process..

[5]  I. Marsic,et al.  Natural communication with information systems , 2000, Proceedings of the IEEE.

[6]  Sadaoki Furui,et al.  A stream-weight optimization method for multi-stream HMMs based on likelihood value normalization , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[7]  Lale Akarun,et al.  Recognizing Two Handed Gestures with Generative, Discriminative and Ensemble Methods Via Fisher Kernels , 2006, MRCS.

[8]  Guido Bologna,et al.  Multimodal tools and interfaces for the intercommunication between visually impaired and "deaf and mute" people , 2006 .

[9]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[10]  Helen Petrie,et al.  Haptic virtual reality for blind computer users , 1998, Assets '98.

[11]  Calle Sjöström,et al.  The IT potential of haptics: Touch access for people with disabilities , 1999 .

[12]  Marjorie Darrah,et al.  Haptic Display of Mathematical Functions for Teaching Mathematics to Students with Vision Disabilities: Design and Proof of Concept , 2000, Haptic Human-Computer Interaction.