An electromyogram-based tapping gesture model with differentiated vibration feedback by low-fidelity actuators

Among other aspects like attention and focus, the effectiveness of actions, such as training in percussion, also relies on the correct gesture and strength of the individual carrying out the activity repetitively to develop proper muscle memory and train motor skills. To make this process economical and efficient, we propose a system that models the feature of direction and strength of a tap and receives the corresponding active haptic feedback in an immersive virtual environment. We propose a novel tapping gesture model that takes electromyogram and vibration feedback into consideration. We also propose an approach of designing vibration modes that follow the mechanical stimulation principles and generate distinguishable vibration feedback with low-fidelity actuators. We developed a prototype using myoelectric and haptic apparatus (Myo armband and HTC VIVE controllers) and evaluated our system by conducting a user study with 50 participants. The evaluation shows that the users can distinguish three common materials (wood, rubber, and aluminum), and our system allows them to actively adjust their direction and strength to grasp the correct point while tapping. We demonstrate the efficacy of our tapping model on a virtual Chinese chimes application and observe positive feedback from both novices and expert users.

[1]  Heung-Ho Choi,et al.  The Development of Muscle Training System Using the Electromyogram and Interactive Game for Physical Rehabilitation , 2011 .

[2]  José Luis Pons Rovira,et al.  Virtual reality training and EMG control of the MANUS hand prosthesis , 2005, Robotica.

[3]  Christopher J. Martin,et al.  Virtual reality simulators: Current status in acquisition and assessment of surgical skills , 2002, ANZ journal of surgery.

[4]  Robert Richer,et al.  Exploring the Feasibility of EMG Based Interaction for Assessing Cognitive Capacity in Virtual Reality , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[5]  S J Bolanowski,et al.  Vibrotactile temporal summation: probability summation or neural integration? , 1999, Somatosensory & motor research.

[6]  Alexander Refsum Jensenius,et al.  Mumyo - evaluating and exploring the myo armband for musical interaction , 2015, NIME.

[7]  Hiroyuki Kajimoto,et al.  Augmentation of Material Property by Modulating Vibration Resulting from Tapping , 2012, EuroHaptics.

[8]  Hiroyuki Shinoda,et al.  A method of selective stimulation to epidermal skin receptors for realistic touch feedback , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[9]  Allison M. Okamura,et al.  Reality-based models for vibration feedback in virtual environments , 2001 .

[10]  P. Lamata,et al.  Tissue consistency perception in laparoscopy to define the level of fidelity in virtual reality simulation , 2006, Surgical Endoscopy And Other Interventional Techniques.

[11]  Satoshi Tadokoro,et al.  A tactile synthesis method using multiple frequency vibrations for representing virtual touch , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  G A Gescheider.  Evidence in support of the duplex theory of mechanoreception. , 1976, Sensory processes.

[13]  Nicola Vitiello,et al.  Intention-Based EMG Control for Powered Exoskeletons , 2012, IEEE Transactions on Biomedical Engineering.

[14]  T. Iijima,et al.  Prediction of four degrees of freedom arm movement using EMG signal , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[15]  R. Johansson,et al.  Responses in glabrous skin mechanoreceptors during precision grip in humans , 2004, Experimental Brain Research.

[16]  Toshio Tsuji,et al.  An MMG-based human-assisting manipulator using acceleration sensors , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[17]  Desney S. Tan,et al.  Making muscle-computer interfaces more practical , 2010, CHI.

[18]  S. Bolanowski,et al.  A four-channel analysis of the tactile sensitivity of the fingertip: frequency selectivity, spatial summation, and temporal summation , 2002, Somatosensory & motor research.

[19]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[20]  P. Ström,et al.  Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents , 2006, Surgical Endoscopy And Other Interventional Techniques.

[21]  Stephen McAdams,et al.  Multisensory integration in percussion performance , 2010 .

[22]  R.N. Scott,et al.  A new strategy for multifunction myoelectric control , 1993, IEEE Transactions on Biomedical Engineering.

[23]  A. Landi.  Human Hand Function , 2007 .

[24]  Kai Kunze,et al.  Assessing hands-free interactions for VR using eye gaze and electromyography , 2018, Virtual Reality.

[25]  Patrick Olivier,et al.  Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor , 2012, UIST.

[26]  Shana Smith,et al.  A virtual reality keyboard with realistic haptic feedback in a fully immersive virtual environment , 2017, Virtual Reality.

[27]  Joris M. Lambrecht,et al.  Electromyogram-based neural network control of transhumeral prostheses. , 2011, Journal of rehabilitation research and development.

[28]  Timothy Bretl,et al.  Prediction of distal arm joint angles from EMG and shoulder orientation for prosthesis control , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[29]  Desney S. Tan,et al.  Enabling always-available input with muscle-computer interfaces , 2009, UIST '09.

[30]  Federico Avanzini,et al.  Integrating physically based sound models in a multimodal rendering architecture , 2006, Comput. Animat. Virtual Worlds.

[31]  Katherine J. Kuchenbecker,et al.  Improving contact realism through event-based haptic feedback , 2006, IEEE Transactions on Visualization and Computer Graphics.

[32]  Federico Fontana,et al.  Implementation and Characterization of Vibrotactile Interfaces , 2018 .

[33]  Claudio Pacchierotti,et al.  Cutaneous Haptic Feedback in Robotic Teleoperation , 2015, Springer Series on Touch and Haptic Systems.

[34]  Seungmoon Choi,et al.  Vibrotactile Display: Perception, Technology, and Applications , 2013, Proceedings of the IEEE.

[35]  T S Kuo,et al.  Real-time implementation of electromyogram pattern recognition as a control command of man-machine interface. , 1996, Medical engineering & physics.

[36]  Hiroyuki Kajimoto,et al.  Wearable tactile device using mechanical and electrical stimulation for fingertip interaction with virtual world , 2017, 2017 IEEE Virtual Reality (VR).

[37]  Jhonatan Camacho Navarro,et al.  EMG-BASED SYSTEM FOR BASIC HAND MOVEMENT RECOGNITION , 2012 .

[38]  Rahman Davoodi,et al.  Prediction of Distal Arm Posture in 3-D Space From Shoulder Movements for Control of Upper Limb Prostheses , 2008, Proceedings of the IEEE.

[39]  Stefania Serafin,et al.  Effect on high versus low fidelity haptic feedback in a virtual reality baseball simulation , 2017, 2017 IEEE Virtual Reality (VR).

[40]  M. Schijven,et al.  The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review , 2009, Surgical Endoscopy.

[41]  A. Phinyomark,et al.  Evaluation of EMG feature extraction for hand movement recognition based on Euclidean distance and standard deviation , 2010, ECTI-CON2010: The 2010 ECTI International Confernce on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology.

[42]  Chongsan Kwon,et al.  Verification of the possibility and effectiveness of experiential learning using HMD-based immersive VR technologies , 2018, Virtual Reality.

[43]  G.M. Lyons,et al.  A computer game-based EMG biofeedback system for muscle rehabilitation , 2003, Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439).