A combination of static and stroke gesture with speech for multimodal interaction in a virtual environment

The computers nowadays are powerful enough to make the virtual reality technology more practical. Virtual reality is expected to have a bright future due to the rise of several virtual reality head mounted device such as low cost Google Cardboard, Samsung Gear VR, Oculus, HTC Vive and many others. Since virtual reality had been growing fast in this few years thus it is necessary to explore new approaches for user to interact with the virtual reality more naturally. It is an important research in HCI to make the interactions with computers to be as natural as the interaction between humans. One of the approaches is to allow user to interact with the virtual reality through gesture and speech input. This paper presents a virtual reality environment with multimodal interaction technique. It allows user to interact with the virtual reality system with the static and stroke hand gesture along with speech. This multimodal interaction technique is able to perform few functions such as select, move, scale, rotate, copy, mirror, delete, check the shape, check and change the colour of an object.

[1]  Harry Bunt,et al.  Multimodal Human-Computer Communication , 1995, Lecture Notes in Computer Science.

[2]  Björn Schuller,et al.  Using multimodal interaction to navigate in arbitrary virtual VRML worlds , 2001, Workshop on Perceptive User Interfaces.

[3]  Mark Billinghurst,et al.  Grasp-Shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[4]  Judy M. Vance,et al.  Virtual reality for assembly methods prototyping: a review , 2011, Virtual Reality.

[5]  Sharon L. Oviatt,et al.  Multimodal Interaction for 2D and 3D Environments , 1999, IEEE Computer Graphics and Applications.

[6]  Sylvain Paris,et al.  6D hands: markerless hand-tracking for computer aided design , 2011, UIST.

[7]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[8]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[9]  William H. Press,et al.  Numerical recipes in C , 2002 .

[10]  Levent Burak Kara,et al.  An Image-Based Trainable Symbol Recognizer for Sketch-Based Interfaces , 2004, AAAI Technical Report.

[11]  Daniel Sonntag,et al.  On-Body IE: A Head-Mounted Multimodal Augmented Reality System for Learning and Recalling Faces , 2013, 2013 9th International Conference on Intelligent Environments.

[12]  Alexander Kulik.  Building on Realism and Magic for Designing 3D Interaction Techniques , 2009, IEEE Computer Graphics and Applications.

[13]  Bernd Fröhlich,et al.  Cubic-Mouse-Based Interaction in Virtual Environments , 2000, IEEE Computer Graphics and Applications.

[14]  M. Schijven,et al.  Systematic review of serious games for medical education and surgical skills training , 2012, The British journal of surgery.

[15]  Gang Ren,et al.  3D Marking menu selection with freehand gestures , 2012, 2012 IEEE Symposium on 3D User Interfaces (3DUI).

[16]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[17]  Woontack Woo,et al.  A usability study of multimodal input in an augmented reality environment , 2013, Virtual Reality.

[18]  Heedong Ko,et al.  An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures , 2006, ICAT.

[19]  Yang Li,et al.  Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes , 2007, UIST.

[20]  A BoltRichard,et al.  Put-that-there , 1980 .

[21]  Zahira Merchant,et al.  Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis , 2014, Comput. Educ..