Understanding freehand gestures: a study of freehand gestural interaction for immersive VR shopping applications

Unlike retail stores, in which the user is forced to be physically present and active during restricted opening hours, online shops may be more convenient, functional and efficient. However, traditional online shops often have a narrow bandwidth for product visualizations and interactive techniques and lack a compelling shopping context. In this paper, we report a study on eliciting user-defined gestures for shopping tasks in an immersive VR (virtual reality) environment. We made a methodological contribution by providing a varied practice for producing more usable freehand gestures than traditional elicitation studies. Using our method, we developed a gesture taxonomy and generated a user-defined gesture set. To validate the usability of the derived gesture set, we conducted a comparative study and answered questions related to the performance, error count, user preference and effort required from end-users to use freehand gestures compared with traditional immersive VR interaction techniques, such as the virtual handle controller and ray-casting techniques. Experimental results show that the freehand-gesture-based interaction technique was rated to be the best in terms of task load, user experience, and presence without the loss of performance (i.e., speed and error count). Based on our findings, we also developed several design guidelines for gestural interaction.

[1]  Andrei Sherstyuk,et al.  Design and Development of a Pose-Based Command Language for Triage Training in Virtual Reality , 2007, 2007 IEEE Symposium on 3D User Interfaces.

[2]  G. Riva,et al.  A virtual reality paradigm for the assessment and rehabilitation of executive function deficits post stroke: Feasibility study , 2008, 2008 Virtual Rehabilitation.

[3]  Michael Rohs,et al.  User-defined gestures for connecting mobile phones, public displays, and tabletops , 2010, Mobile HCI.

[4]  Xiaolong Zhang,et al.  User-defined gesture interaction for immersive VR shopping applications , 2018, Behav. Inf. Technol..

[5]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[6]  L.S. Cardoso,et al.  Using Virtual Environments for Stroke Rehabilitation , 2006, 2006 International Workshop on Virtual Rehabilitation.

[7]  Yi Li,et al.  Real-time oriented behavior-driven 3D freehand tracking for direct interaction , 2013, Pattern Recognit..

[8]  Ning Zhang,et al.  The virtual reality system in electronic commerce , 2012, 2012 IEEE Symposium on Robotics and Applications (ISRA).

[9]  Sriram Subramanian,et al.  Would you do that?: understanding social acceptance of gestural interfaces , 2010, Mobile HCI.

[10]  Bongshin Lee,et al.  Reducing legacy bias in gesture elicitation studies , 2014, INTR.

[11]  Hai-Ning Liang,et al.  User-elicited dual-hand interactions for manipulating 3D objects in virtual reality environments , 2018, Human-centric Computing and Information Sciences.

[12]  Gang Ren,et al.  3D selection with freehand gesture , 2013, Comput. Graph..

[13]  Sangil Choi,et al.  Understanding people with human activities and social interactions for human-centered computing , 2016, Human-centric Computing and Information Sciences.

[14]  Peng Song,et al.  A handle bar metaphor for virtual object manipulation with mid-air interaction , 2012, CHI.

[15]  Gerard Jounghyun Kim,et al.  IoT + AR: pervasive and augmented environments for “Digi-log” shopping experience , 2019, Human-centric Computing and Information Sciences.

[16]  Katsumi Yamashita,et al.  Mixed-reality shopping system using HMD and smartwatch , 2015, UbiComp/ISWC Adjunct.

[17]  Stephan Lukosch,et al.  On the Efficiency of a VR Hand Gesture-Based Interface for 3D Object Manipulations in Conceptual Design , 2017, Int. J. Hum. Comput. Interact..

[18]  Tobias Höllerer,et al.  Vision-based interfaces for mobility , 2004, The First Annual International Conference on Mobile and Ubiquitous Systems: Networking and Services, 2004. MOBIQUITOUS 2004..

[19]  Antonio Krüger,et al.  A virtual reality shopping experience using the apartment metaphor , 2018, AVI.

[20]  João Marcelo X. N. Teixeira,et al.  A comparative evaluation of direct hand and wand interactions on consumer devices , 2018, Comput. Graph..

[21]  Jean Vanderdonckt,et al.  Gestures for Smart Rings: Empirical Results, Insights, and Design Implications , 2018, Conference on Designing Interactive Systems.

[22]  Eulalie Verhulst,et al.  3D Interaction Techniques for Virtual Shopping: Design and Preliminary Study , 2016, VISIGRAPP.

[23]  Evelyne Klinger,et al.  Using the virtual action planning-supermarket for evaluating executive functions in people with stroke. , 2014, Journal of stroke and cerebrovascular diseases : the official journal of National Stroke Association.

[24]  Christina Boucher,et al.  Exploring Non-touchscreen Gestures for Smartwatches , 2016, CHI.

[25]  Eva Hornecker,et al.  Modifying Gesture Elicitation: Do Kinaesthetic Priming and Increased Production Reduce Legacy Bias? , 2016, Tangible and Embedded Interaction.

[26]  Tian Chen,et al.  EasyMall - An Interactive Virtual Shopping System , 2008, 2008 Fifth International Conference on Fuzzy Systems and Knowledge Discovery.

[27]  Holger Regenbrecht,et al.  Real and Illusory Interactions Enhance Presence in Virtual Environments , 2002, Presence: Teleoperators & Virtual Environments.

[28]  Alexei Sourin,et al.  Mid-air interaction with optical tracking for 3D modeling , 2018, Comput. Graph..

[29]  R. Kizony,et al.  Performance within a virtual supermarket and its relationship to executive functions in post-stroke patients , 2006, 2006 International Workshop on Virtual Rehabilitation.

[30]  Martin Hitz,et al.  Exploring intuitiveness of metaphor-based gestures for UAV navigation , 2017, 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[31]  Vivek K. Goyal,et al.  Mime: compact, low power 3D gesture sensing for interaction with head mounted displays , 2013, UIST.

[32]  Radu-Daniel Vatavu,et al.  Formalizing Agreement Analysis for Elicitation Studies: New Measures, Significance Test, and Toolkit , 2015, CHI.

[33]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[34]  Andrea Sanna,et al.  3D-dvshop: a 3D dynamic virtual shop , 2002 .

[35]  Ying-Chao Tung,et al.  User-Defined Game Input for Smart Glasses in Public Space , 2015, CHI.

[36]  Teddy Seyed,et al.  Eliciting usable gestures for multi-display environments , 2012, ITS.

[37]  Xiaolong Zhang,et al.  Combining hidden Markov model and fuzzy neural network for continuous recognition of complex dynamic gestures , 2015, The Visual Computer.

[38]  Antonio Krüger,et al.  VRShop: A Mobile Interactive Virtual Reality Shopping Environment Combining the Benefits of On- and Offline Shopping , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[39]  Jianmin Wang,et al.  User-centered gesture development in TV viewing environment , 2014, Multimedia Tools and Applications.

[40]  Jianmin Wang,et al.  A visual attention-based method to address the midas touch problem existing in gesture-based interaction , 2014, The Visual Computer.

[41]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[42]  Adrian David Cheok,et al.  22nd International Conference on Human-Computer Interaction with Mobile Devices and Services , 2007, Lecture Notes in Computer Science.

[43]  Yang Li,et al.  User-defined motion gestures for mobile interaction , 2011, CHI.

[44]  David K. Harrison,et al.  Development and Heuristic Evaluation of Semi-immersive Hand-Gestural Virtual Reality Interface for Luxury Brands Online Stores , 2017, AVR.

[45]  Andrea Sanna,et al.  3D-dvshop: a 3D Dynamic Virtual Shop , 2001, Eurographics Multimedia Workshop.

[46]  Michael S. Horn,et al.  Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces , 2012, ITS 2013.

[47]  Teddy Seyed,et al.  User Elicitation on Single-hand Microgestures , 2016, CHI.

[48]  Jun Wang,et al.  Indoor scene modeling from a single image using normal inference and edge features , 2017, The Visual Computer.

[49]  David Lundberg,et al.  Shopping Using Gesture-Driven Interaction , 2015 .

[50]  Huiyue Wu,et al.  The Gesture Disagreement Problem in Free-hand Gesture Interaction , 2018, Int. J. Hum. Comput. Interact..

[51]  Susan T. Dumais,et al.  The vocabulary problem in human-system communication , 1987, CACM.

[52]  Meredith Ringel Morris,et al.  Understanding users' preferences for surface gestures , 2010, Graphics Interface.

[53]  Konrad Tollmar,et al.  Navigating in virtual environments using a vision-based interface , 2004, NordiCHI '04.

[54]  Ci Wang,et al.  User-Defined Gestures for Gestural Interaction: Extending from Hands to Other Body Parts , 2018, Int. J. Hum. Comput. Interact..