Kinect-based visual communication system

Nowadays, most existing online instant messaging tools, such as Live Messenger, Google Talk, Yahoo Messenger, ICQ, enable people to communicate with each other no matter where and when they are. However, it is still difficult for people who speak different native languages and do not understand each other's to communicate smoothly. It could be more difficult when people with hearing impairment are trying to use those tools. Moreover, users' hands are usually tied up with keyboard and mouse to keep typing messages. To deal with these disadvantages, we design a Kinect-based Visual Communication System (KVCS), which contains following features: (1) Kinect-based sign language recognition module to make deaf-mute persons be able to chat. (2) Kinect-based expression recognition module to enrich online chatting experiences at the same time.(3) Kinect-based speech recognition module to free users' hands when chatting. (4) Cross-media multi-lingual visualized translation module to enable users to catch the meanings of conversation much easier. Experiments on our system demonstrate that this novel KVCS provides a powerful and efficient communication function and a wonderful user experience.

[1]  Wen Gao,et al.  Large vocabulary sign language recognition based on hierarchical decision trees , 2003, ICMI '03.

[2]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[3]  James Ze Wang,et al.  The Story Picturing Engine---a system for automatic text illustration , 2006, TOMCCAP.

[4]  Xiaojin Zhu,et al.  A Text-to-Picture Synthesis System for Augmenting Communication , 2007, AAAI.

[5]  Hermann Hienz,et al.  Video-based continuous sign language recognition using statistical methods , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[6]  Ying-li Tian,et al.  Evaluation of Face Resolution for Expression Analysis , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[7]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[8]  Xiaojin Zhu,et al.  Easy as ABC? Facilitating Pictorial Communication via Semantically Enhanced Layout , 2008, CoNLL.

[9]  Paul Lukowicz,et al.  Using multiple sensors for mobile sign language recognition , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..

[10]  Masaru Takeuchi,et al.  A method for recognizing a sequence of sign language words represented in a Japanese sign language sentence , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[11]  Robert W. Lindeman,et al.  A new instrumented approach for translating American Sign Language into sound and text , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[12]  Lisa M. Brown,et al.  Real World Real-time Automatic Recognition of Facial Expressions , 2003 .

[13]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[14]  Changsheng Xu,et al.  Chat with illustration: a chat system with visual aids , 2012, ICIMCS '12.

[15]  Nicolas Pugeault,et al.  Spelling it out: Real-time ASL fingerspelling recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).