A 3D interactive kiosk system

One of the long-term goals in human-computer interaction is to utilize more intuitive and natural methods such as speech and hand gesture that a user would employ for communication. In this paper, we present a multi-modal 3D interaction mechanism, in which user can interact with a 3D model of a tourist location displayed on the kiosk screen from a one meter distance by means of gestures and voice commands without wearing any special device in a public place, with a complex and non-static background environment. The system can be used in many applications such entertainment, touring, education, museum displays, and advertising.

[1]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[2]  Yuan Yao,et al.  Hand tracking in time-varying illumination , 2004, Proceedings of 2004 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.04EX826).

[3]  Mohammed Yeasin,et al.  A real-time framework for natural multimodal interaction with large screen displays , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[4]  Michael G. Strintzis,et al.  Masterpiece: physical interaction and 3D content-based search in VR applications , 2006, IEEE MultiMedia.

[5]  Matheen Siddiqui,et al.  Robust real-time upper body limb detection and tracking , 2006, VSSN '06.

[6]  Steve Mann,et al.  Camera response function recovery from different illuminations of identical subject matter , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[7]  Marc Erich Latoschik.  A user interface framework for multimodal VR interactions , 2005, ICMI '05.

[8]  Edward Cutrell,et al.  FlowMouse: A Computer Vision-Based Pointing and Gesture Input Device , 2005, INTERACT.

[9]  Maribeth Gandy Coleman,et al.  The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[10]  William Ribarsky,et al.  Speech and Gesture Multimodal Control of a Whole Earth 3D Visualization Environment , 2002, VisSym.

[11]  Haizhou Li,et al.  Language identification through large vocabulary continuous speech recognition , 2004, 2004 International Symposium on Chinese Spoken Language Processing.

[12]  Alexander H. Waibel,et al.  Segmenting hands of arbitrary color , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[13]  Yacine Bellik,et al.  A framework to manage multimodal fusion of events for advanced interactions within Virtual Environments , 2002, EGVE.

[14]  Marcelo Knörich Zuffo,et al.  On the usability of gesture interfaces in virtual reality environments , 2005, CLIHC '05.

[15]  S. Mann,et al.  Digital Camera Sensor Noise Estimation from Different Illuminations of Identical Subject Matter , 2005, 2005 5th International Conference on Information Communications & Signal Processing.

[16]  J. Ohya,et al.  Automatic skin-color distribution extraction for face detection and tracking , 2000, WCC 2000 - ICSP 2000. 2000 5th International Conference on Signal Processing Proceedings. 16th World Computer Congress 2000.

[17]  Thomas B. Moeslund,et al.  Vision-Based User Interface for Interacting with a Virtual Environment , 2000 .

[18]  Rainer Stiefelhagen,et al.  Pointing gesture recognition based on 3D-tracking of face, hands and head orientation , 2003, ICMI '03.

[19]  Kwang-Ting Cheng,et al.  Adaptive learning of an accurate skin-color model , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[20]  Nikolaos G. Bourbakis,et al.  A survey of skin-color modeling and detection methods , 2007, Pattern Recognit..

[21]  Nuria Oliver,et al.  GWINDOWS: Towards Robust Perception-Based UI , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[22]  Gary R. Bradski,et al.  Real time face and object tracking as a component of a perceptual user interface , 1998, Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201).