Using Hand Gesture and Speech in a Multimodal Augmented Reality Environment

In this work we describe a 3D authoring tool which takes advantage of multimodal interfaces such as gestures and speech. This tool allows real-time Augmented Reality aimed to aid the tasks of interior architects and designers. This approach intends to be an alternative to traditional techniques. The main benefit of using a multimodal based augmented reality system is the provision of a more transparent, flexible, efficient and expressive means of human-computer interaction.

[1]  Marie-Odile Berger,et al.  Reconstructing while registering: a novel approach for markerless augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[2]  Andreas Paepcke,et al.  Cooperative gestures: multi-user gestural interactions for co-located groupware , 2006, CHI.

[3]  R. Bastos,et al.  Developing and authoring mixed reality with MX toolkit , 2003, 2003 IEEE International Augmented Reality Toolkit Workshop.

[4]  BillinghurstMark.  Put that where? voice and gesture at the graphics interface , 1998 .

[5]  Antonio Camurri,et al.  Analysis of Expressive Gesture: The EyesWeb Expressive Gesture Processing Library , 2003, Gesture Workshop.

[6]  José Miguel Salles Dias,et al.  Fully automated texture tracking based on natural features extraction and template matching , 2005, ACE '05.

[7]  José Miguel Salles Dias,et al.  O.G.R.E. - Open Gestures Recognition Engine, a Platform for Gesture-Based Communication and Interaction , 2005, Gesture Workshop.

[8]  Sharon Oviatt,et al.  Multimodal Interfaces , 2008, Encyclopedia of Multimedia.

[9]  Patrick Brézillon,et al.  Lecture Notes in Artificial Intelligence , 1999 .

[10]  Jakub Segen,et al.  Look ma, no mouse! , 2000, CACM.

[11]  Antonio Camurri,et al.  Gesture-Based Communication in Human-Computer Interaction , 2003, Lecture Notes in Computer Science.

[12]  Sharon L. Oviatt,et al.  Error resolution during multimodal human-computer interaction , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[13]  Joaquim A. Jorge,et al.  Using fuzzy logic to recognize geometric shapes interactively , 2000, Ninth IEEE International Conference on Fuzzy Systems. FUZZ- IEEE 2000 (Cat. No.00CH37063).

[14]  Nicolas Courty,et al.  Gesture in Human-Computer Interaction and Simulation , 2006 .

[15]  Larry S. Davis,et al.  Learning dynamics for exemplar-based gesture recognition , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[16]  J. Jacko,et al.  The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications , 2002 .

[17]  Gaojin Wen,et al.  From motion capture data to character animation , 2006, VRST '06.

[18]  Jakub Segen,et al.  Look ma, no mouse! Simplifying human-computer interaction by using hand gesture. , 2000 .

[19]  Gunther Heidemann,et al.  Multimodal interaction in an augmented reality scenario , 2004, ICMI '04.

[20]  Joaquim A. Jorge,et al.  PTrack: Introducing a Novel Iterative Geometric Pose Estimation for a Marker-based Single Camera Tracking System , 2006, IEEE Virtual Reality Conference (VR 2006).

[21]  Andrew W. Fitzgibbon,et al.  Markerless tracking using planar structures in the scene , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[22]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[23]  Hirokazu Kato,et al.  A registration method based on texture tracking using ARToolKit , 2003, 2003 IEEE International Augmented Reality Toolkit Workshop.

[24]  Thomas B. Moeslund,et al.  Computer vision-based gesture recognition for an augmented reality interface , 2004 .

[25]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[26]  Yu Li,et al.  Magic Story Cube: an interactive tangible interface for storytelling , 2004, ACE '04.

[27]  Gregory D. Abowd,et al.  Charting past, present, and future research in ubiquitous computing , 2000, TCHI.

[28]  Neff Walker,et al.  Evaluation of the CyberGlove as a whole-hand input device , 1995, TCHI.