Multimodal User Interface for Augmented Assembly

In this paper, a multimodal system for augmented reality aided assembly work is designed and implemented. The multimodal interface allows for speech and gestural input. The system emulates a simplified assembly task in a factory. A 3D puzzle is used to study how to implement the augmented assembly system to a real setting in a factory. The system is used as a demonstrator and as a test-bed to evaluate different input modalities for augmented assembly setups. Preliminary system evaluation results are presented, the user experience is discussed, and some directions for future work are given.

[1]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[2]  Jennifer Healey,et al.  Augmented Reality through Wearable Computing , 1997, Presence: Teleoperators & Virtual Environments.

[3]  Bruce H. Thomas,et al.  Emerging Technologies of Augmented Reality , 2006 .

[4]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[5]  Chin-Hui Lee,et al.  DESIGN PRINCIPLES AND TOOLS FOR MULTIMODAL DIALOG SYSTEMS , 2000 .

[6]  Sharon L. Oviatt,et al.  Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions , 2000, Hum. Comput. Interact..

[7]  Sharon L. Oviatt,et al.  Designing robust multimodal systems for universal access , 2001, WUAUC'01.

[8]  Ramesh Raskar,et al.  Modern approaches to augmented reality , 2005, SIGGRAPH Courses.

[9]  Ronald Azuma,et al.  A survey of augmented reality" Presence: Teleoperators and virtual environments , 1997 .

[10]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[11]  Kadri Hacioglu,et al.  Recent improvements in the CU Sonic ASR system for noisy speech: the SPINE task , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[12]  Charles Woodward,et al.  Interactive outdoor mobile augmentation using markerless tracking and GPS , 2007 .

[13]  Michael Rohs,et al.  Marker-Based Embodied Interaction for Handheld Augmented Reality Games , 2007, J. Virtual Real. Broadcast..

[14]  Mark Billinghurst,et al.  Virtual object manipulation using a mobile phone , 2005, ICAT '05.

[15]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.