User-defined gesture interaction for immersive VR shopping applications

ABSTRACT Gesture elicitation studies, which are a popular technology for collecting requirements and expectations by involving real users in gesture design processes, often suffer from gesture disagreement and legacy bias and may not generate optimal gestures for a target system in practice. This paper reports a research project on user-defined gestures for interacting with immersive VR shopping applications. The main contribution of this work is the proposal of a more practical method for deriving more reliable gestures than traditional gesture elicitation studies. We applied this method to a VR shopping application and obtained empirical evidence for the benefits of deriving two gestures in the a priori stage and selecting the top-two gestures in the a posteriori stage of traditional elicitation studies for each referent. We hope that this research can help lay a theoretical foundation for freehand-gesture-based user interface design and be generalised to all freehand-gesture-based applications.

[1]  Ying-Chao Tung,et al.  User-Defined Game Input for Smart Glasses in Public Space , 2015, CHI.

[2]  Elena Mugellini,et al.  Gesturing on the Steering Wheel: a User-elicited taxonomy , 2014, AutomotiveUI.

[3]  Niels Henze,et al.  User-centred process for the definition of free-hand gestures applied to controlling music playback , 2012, Multimedia Systems.

[4]  Meredith Ringel Morris,et al.  Understanding users' preferences for surface gestures , 2010, Graphics Interface.

[5]  Michael Rohs,et al.  User-defined gestures for connecting mobile phones, public displays, and tabletops , 2010, Mobile HCI.

[6]  Frank E. Ritter,et al.  Foundations for Designing User-Centered Systems: What System Designers Need to Know about People , 2014 .

[7]  Teddy Seyed,et al.  Eliciting usable gestures for multi-display environments , 2012, ITS.

[8]  Ling Shao,et al.  RGB-D datasets using microsoft kinect or similar sensors: a survey , 2017, Multimedia Tools and Applications.

[9]  Sriram Subramanian,et al.  Would you do that?: understanding social acceptance of gestural interfaces , 2010, Mobile HCI.

[10]  Giuseppe Riva,et al.  A NeuroVR based tool for cognitive assessment and rehabilitation of post-stroke patients: two case studies , 2009, Annual Review of Cybertherapy and Telemedicine.

[11]  Martin Hitz,et al.  Coherence evaluation of input vocabularies to enhance usability and user experience , 2017, EICS.

[12]  Albrecht Schmidt,et al.  Gestural interaction on the steering wheel: reducing the visual demand , 2011, CHI.

[13]  Martin Hitz,et al.  Exploring intuitiveness of metaphor-based gestures for UAV navigation , 2017, 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[14]  Konrad Tollmar,et al.  Navigating in virtual environments using a vision-based interface , 2004, NordiCHI '04.

[15]  Mark Billinghurst,et al.  User Defined Gestures for Augmented Virtual Mirrors: A Guessability Study , 2015, CHI Extended Abstracts.

[16]  Evelyne Klinger,et al.  Virtual Action Planning in Parkinson's Disease: AControl Study , 2006, Cyberpsychology Behav. Soc. Netw..

[17]  Andy Cockburn,et al.  User-defined gestures for augmented reality , 2013, INTERACT.

[18]  David Lundberg,et al.  Shopping Using Gesture-Driven Interaction , 2015 .

[19]  Evelyne Klinger,et al.  Using the virtual action planning-supermarket for evaluating executive functions in people with stroke. , 2014, Journal of stroke and cerebrovascular diseases : the official journal of National Stroke Association.

[20]  Hanseok Ko,et al.  Gesture recognition using depth-based hand tracking for contactless controller application , 2012, 2012 IEEE International Conference on Consumer Electronics (ICCE).

[21]  Lu Yang,et al.  Survey on 3D Hand Gesture Recognition , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[22]  L.S. Cardoso,et al.  Using Virtual Environments for Stroke Rehabilitation , 2006, 2006 International Workshop on Virtual Rehabilitation.

[23]  Kris Luyten,et al.  Multi-viewer gesture-based interaction for omni-directional video , 2014, CHI.

[24]  Christina Boucher,et al.  Exploring Non-touchscreen Gestures for Smartwatches , 2016, CHI.

[25]  Joseph J. LaViola,et al.  Exploring the usefulness of finger-based 3D gesture menu selection , 2014, CHI.

[26]  Joseph J. LaViola,et al.  Towards user-defined multi-touch gestures for 3D objects , 2013, ITS.

[27]  Thomas B. Moeslund,et al.  A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI , 2003, Gesture Workshop.

[28]  Anupam Agrawal,et al.  Vision based hand gesture recognition for human computer interaction: a survey , 2012, Artificial Intelligence Review.

[29]  Meredith Ringel Morris,et al.  Web on the wall: insights from a multimodal interaction elicitation study , 2012, ITS.

[30]  Bongshin Lee,et al.  Reducing legacy bias in gesture elicitation studies , 2014, INTR.

[31]  Joseph J. LaViola,et al.  Exploring 3d gesture metaphors for interaction with unmanned aerial vehicles , 2013, IUI '13.

[32]  Peng Song,et al.  A handle bar metaphor for virtual object manipulation with mid-air interaction , 2012, CHI.

[33]  Radu-Daniel Vatavu,et al.  Formalizing Agreement Analysis for Elicitation Studies: New Measures, Significance Test, and Toolkit , 2015, CHI.

[34]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[35]  Mohammad Obaid,et al.  User-Defined Body Gestures for Navigational Control of a Humanoid Robot , 2012, ICSR.

[36]  Evelyne Klinger,et al.  Using virtual reality to evaluate executive functioning among persons with schizophrenia: A validity study , 2009, Schizophrenia Research.

[37]  M. Levin,et al.  Rehabilitation tools along the reality continuum : from mock-up to virtual interactive shopping to a living lab , 2012 .

[38]  David Ott,et al.  Web on the Wall Reloaded: Implementation, Replication and Refinement of User-Defined Interaction Sets , 2014, ITS '14.

[39]  Ali Mazalek,et al.  Exploring the design space of gestural interaction with active tokens through user-defined gestures , 2014, CHI.

[40]  Stephan Lukosch,et al.  On the Efficiency of a VR Hand Gesture-Based Interface for 3D Object Manipulations in Conceptual Design , 2017, Int. J. Hum. Comput. Interact..

[41]  Tobias Höllerer,et al.  Vision-based interfaces for mobility , 2004, The First Annual International Conference on Mobile and Ubiquitous Systems: Networking and Services, 2004. MOBIQUITOUS 2004..

[42]  Jianmin Wang,et al.  User-centered gesture development in TV viewing environment , 2014, Multimedia Tools and Applications.

[43]  Per Ola Kristensson,et al.  Memorability of pre-designed and user-defined gesture sets , 2013, CHI.

[44]  Per Ola Kristensson,et al.  Memorability of pre-designed and user-defined gesture sets , 2013, CHI.

[45]  Yang Li,et al.  User-defined motion gestures for mobile interaction , 2011, CHI.

[46]  David K. Harrison,et al.  Development and Heuristic Evaluation of Semi-immersive Hand-Gestural Virtual Reality Interface for Luxury Brands Online Stores , 2017, AVR.

[47]  Shin'ichi Satoh,et al.  Human gesture recognition system for TV viewing using time-of-flight camera , 2011, Multimedia Tools and Applications.

[48]  Seong-Whan Lee,et al.  Gesture Spotting and Recognition for Human–Robot Interaction , 2007, IEEE Transactions on Robotics.

[49]  Jean Vanderdonckt,et al.  Gestures for Smart Rings: Empirical Results, Insights, and Design Implications , 2018, Conference on Designing Interactive Systems.

[50]  Eulalie Verhulst,et al.  3D Interaction Techniques for Virtual Shopping: Design and Preliminary Study , 2016, VISIGRAPP.

[51]  Elisabeth André,et al.  Studying user-defined iPad gestures for interaction in multi-display environment , 2012, IUI '12.

[52]  Huiyue Wu,et al.  The Gesture Disagreement Problem in Free-hand Gesture Interaction , 2018, Int. J. Hum. Comput. Interact..

[53]  Donghun Lee,et al.  Towards successful user interaction with systems: focusing on user-derived gestures for smart home systems. , 2014, Applied ergonomics.

[54]  Ci Wang,et al.  User-Defined Gestures for Gestural Interaction: Extending from Hands to Other Body Parts , 2018, Int. J. Hum. Comput. Interact..

[55]  Abdulmotaleb El-Saddik,et al.  An Elicitation Study on Gesture Preferences and Memorability Toward a Practical Hand-Gesture Vocabulary for Smart Televisions , 2015, IEEE Access.

[56]  Frank E. Ritter,et al.  Foundations for Designing User-Centered Systems , 2014, Springer London.

[57]  Patrice L Tamar Weiss,et al.  Training multitasking in a virtual supermarket: a novel intervention after stroke. , 2009, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[58]  Radu-Daniel Vatavu,et al.  User-defined gestures for free-hand TV control , 2012, EuroITV.

[59]  Eva Hornecker,et al.  Modifying Gesture Elicitation: Do Kinaesthetic Priming and Increased Production Reduce Legacy Bias? , 2016, Tangible and Embedded Interaction.

[60]  Anne Marie Piper,et al.  A Wizard-of-Oz elicitation study examining child-defined gestures with a whole-body interface , 2013, IDC.

[61]  Brad A. Myers,et al.  Maximizing the guessability of symbolic input , 2005, CHI Extended Abstracts.

[62]  Preben Hansen,et al.  Effects of User’s Hand Orientation and Spatial Movements on Free Hand Interactions with Large Displays , 2018, Int. J. Hum. Comput. Interact..

[63]  Andrei Sherstyuk,et al.  Design and Development of a Pose-Based Command Language for Triage Training in Virtual Reality , 2007, 2007 IEEE Symposium on 3D User Interfaces.

[64]  Wendy Yee.  Potential Limitations of Multi-touch Gesture Vocabulary: Differentiation, Adoption, Fatigue , 2009, HCI.

[65]  Per Ola Kristensson,et al.  User-defined Interface Gestures: Dataset and Analysis , 2014, ITS '14.

[66]  Sebastian Möller,et al.  I'm home: Defining and evaluating a gesture set for smart-home control , 2011, Int. J. Hum. Comput. Stud..

[67]  Yi Li,et al.  Real-time oriented behavior-driven 3D freehand tracking for direct interaction , 2013, Pattern Recognit..

[68]  Teddy Seyed,et al.  User Elicitation on Single-hand Microgestures , 2016, CHI.

[69]  Radu-Daniel Vatavu,et al.  On free-hand TV control: experimental results on user-elicited gestures with Leap Motion , 2015, Personal and Ubiquitous Computing.