DextrEMS: Increasing Dexterity in Electrical Muscle Stimulation by Combining it with Brakes

Electrical muscle stimulation (EMS) is an emergent technique that miniaturizes force feedback, especially popular for untethered haptic devices, such as mobile gaming, VR, or AR. However, the actuation displayed by interactive systems based on EMS is coarse and imprecise. EMS systems mostly focus on inducing movements in large muscle groups such as legs, arms, and wrists; whereas individual finger poses, which would be required, for example, to actuate a user's fingers to fingerspell even the simplest letters in sign language, are not possible. The lack of dexterity in EMS stems from two fundamental limitations: (1) lack of independence: when a particular finger is actuated by EMS, the current runs through nearby muscles, causing unwanted actuation of adjacent fingers; and, (2) unwanted oscillations: while it is relatively easy for EMS to start moving a finger, it is very hard for EMS to stop and hold that finger at a precise angle; because, to stop a finger, virtually all EMS systems contract the opposing muscle, typically achieved via controllers (e.g., PID)—unfortunately, even with the best controller tuning, this often results in unwanted oscillations. To tackle these limitations, we propose dextrEMS, an EMS-based haptic device featuring mechanical brakes attached to each finger joint. The key idea behind dextrEMS is that while the EMS actuates the fingers, it is our mechanical brake that stops the finger in a precise position. Moreover, it is also the brakes that allow dextrEMS to select which fingers are moved by EMS, eliminating unwanted movements by preventing adjacent fingers from moving. We implemented dextrEMS as an untethered haptic device, weighing only 68g, that actuates eight finger joints independently (metacarpophalangeal and proximal interphalangeal joints for four fingers), which we demonstrate in a wide range of haptic applications, such as assisted fingerspelling, a piano tutorial, guitar tutorial, and a VR game. Finally, in our technical evaluation, we found that dextrEMS outperformed EMS alone by doubling its independence and reducing unwanted oscillations.

[1]  Jin-Shin Lai,et al.  Development of the FES system with neural network + PID controller for the stroke , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[2]  M. Popovic,et al.  PID Controller Design for FES Applied to Ankle Muscles in Neuroprosthesis for Standing Balance , 2017, Front. Neurosci..

[3]  Grigore C. Burdea,et al.  The Rutgers Master II-new design force-feedback glove , 2002 .

[4]  Toshio Tsuji,et al.  Stiffness Display by Muscle Contraction Via Electric Muscle Stimulation , 2016, IEEE Robotics and Automation Letters.

[5]  Pedro Lopes,et al.  Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation , 2018, CHI.

[6]  Craig R. Carignan,et al.  Development of an exoskeleton haptic interface for virtual task training , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[7]  Jeremy R. Cooperstock,et al.  Raising the Heat: Electrical Muscle Stimulation for Simulated Heat Withdrawal Response , 2017, UIST.

[8]  Florian Alt,et al.  Supporting interaction in public space with electrical muscle stimulation , 2013, UbiComp.

[9]  Stefan Schneegaß,et al.  EMS in HCI: challenges and opportunities in actuating human bodies , 2017, MobileHCI.

[10]  Massimo Bergamasco,et al.  An arm exoskeleton system for teleoperation and virtual environments applications , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[11]  M. Popovic,et al.  Multi-field surface electrode for selective electrical stimulation. , 2005, Artificial organs.

[12]  Jun Rekimoto,et al.  PossessedHand: techniques for controlling human hands using electrical muscles stimuli , 2011, CHI.

[13]  Wit Wannasuphoprasit,et al.  Design of a novel finger exoskeleton with a sliding six-bar joint mechanism , 2015, AH.

[14]  Michael Rohs,et al.  Emotion Actuator: Embodied Emotional Feedback through Electroencephalography and Electrical Muscle Stimulation , 2017, CHI.

[15]  Marc H Schieber,et al.  Human finger independence: limitations due to passive mechanical coupling versus active neuromuscular control. , 2004, Journal of neurophysiology.

[16]  Xiaoying Lü,et al.  Electrode placement on the forearm for selective stimulation of finger extension/flexion , 2018, PloS one.

[17]  Michael Rohs,et al.  Cruise Control for Pedestrians: Controlling Walking Direction using Electrical Muscle Stimulation , 2015, CHI.

[18]  Sebastian O. H. Madgwick,et al.  An efficient orientation filter for inertial and inertial / magnetic sensor arrays , 2010 .

[19]  Alex Olwal,et al.  Grabity: A Wearable Haptic Interface for Simulating Weight and Grasping in Virtual Reality , 2017, UIST.

[20]  Michael Rohs,et al.  MuscleIO: Muscle-Based Input and Output for Casual Notifications , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[21]  Akira Ishii,et al.  Stimulated percussions: method to control human for learning music by using electrical muscle stimulation , 2017, AH.

[22]  Tomohiko Hayakawa,et al.  Comparison of reaction times in response to electrical and Visual Stimulation using a high-speed camera , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[23]  Pedro Lopes,et al.  Immense Power in a Tiny Package: Wearables Based on Electrical Muscle Stimulation , 2017, IEEE Pervasive Computing.

[24]  Yasuaki Kakehi,et al.  Paralogue: A Remote Conversation System Using a Hand Avatar which Postures are Controlled with Electrical Muscle Stimulation , 2018, AH.

[25]  Dieter Schmalstieg,et al.  Using neuromuscular electrical stimulation for pseudo-haptic feedback , 2006, VRST '06.

[26]  Pedro Lopes,et al.  Preemptive Action: Accelerating Human Reaction using Electrical Muscle Stimulation Without Compromising Agency , 2019, CHI.

[27]  Sebastian Boring,et al.  Automatic Calibration of High Density Electric Muscle Stimulation , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[28]  Anna Vallgårda,et al.  FeltRadio: Sensing and Making Sense of Wireless Traffic , 2016, Conference on Designing Interactive Systems.

[29]  Michael Rohs,et al.  Follow the Force: Steering the Index Finger towards Targets using EMS , 2016, CHI Extended Abstracts.

[30]  Massimo Bergamasco,et al.  On the Integration of Tactile and Force Feedback , 2012 .

[31]  Yoichi Ochiai,et al.  Building a feedback loop between electrical stimulation and percussion learning , 2018, SIGGRAPH 2018.

[32]  Stefan Schneegaß,et al.  Embodied notifications: implicit notifications through electrical muscle stimulation , 2016, MobileHCI Adjunct.

[33]  A. Kralj,et al.  Programmed Six-Channel Electrical Stimulator for Complex Stimulation of Leg Muscles During Walking , 1979, IEEE Transactions on Biomedical Engineering.

[34]  Carlo Alberto Avizzano,et al.  Design and Development of a Hand Exoskeleton Robot for Active and Passive Rehabilitation , 2016 .

[35]  Igor Zubrycki,et al.  Novel Haptic Device Using Jamming Principle for Providing Kinaesthetic Feedback in Glove-Based Control Interface , 2016, Journal of Intelligent & Robotic Systems.

[36]  Max Pfeiffer,et al.  3D Virtual Hand Selection with EMS and Vibration Feedback , 2015, CHI Extended Abstracts.

[37]  Kasper Hornbæk,et al.  Wanding Through Space: Interactive Calibration for Electric Muscle Stimulation , 2018, AH.

[38]  Pedro Lopes,et al.  Action-Dependent Processing of Touch in the Human Parietal Operculum and Posterior Insula. , 2019, Cerebral cortex.

[39]  Sean Follmer,et al.  A Soft, Controllable, High Force Density Linear Brake Utilizing Layer Jamming , 2018, IEEE Robotics and Automation Letters.

[40]  H Hummelsheim,et al.  The functional value of electrical muscle stimulation for the rehabilitation of the hand in stroke patients. , 1997, Scandinavian journal of rehabilitation medicine.

[41]  Pinhas Ben-Tzvi,et al.  RML Glove—An Exoskeleton Glove Mechanism With Haptics Feedback , 2015, IEEE/ASME Transactions on Mechatronics.

[42]  Dejan Milutinovic,et al.  Unilateral and Bilateral Rehabilitation of the Upper Limb Following Stroke via an Exoskeleton , 2014 .

[43]  Emi Tamaki,et al.  UnlimitedHand: Input and Output Hand Gestures with Less Calibration Time , 2016, UIST.

[44]  Farzam Farbiz,et al.  An electrical muscle stimulation haptic feedback for mixed reality tennis game , 2007, SIGGRAPH '07.

[45]  Michael Rohs,et al.  A Wearable Force Feedback Toolkit with Electrical Muscle Stimulation , 2016, CHI Extended Abstracts.

[46]  Kai Kunze,et al.  EMS icons: conveying information by analogy to enhance communication through electrical muscle stimulation , 2017, UbiComp/ISWC Adjunct.

[47]  Sean Follmer,et al.  Wolverine: A Wearable Haptic Interface for Grasping in VR , 2016, UIST.

[48]  Kenji Suzuki,et al.  Wired muscle: generating faster kinesthetic reaction by inter-personally connecting muscles , 2017, SIGGRAPH Emerging Technologies.

[49]  Pedro Lopes,et al.  Muscle-propelled force feedback: bringing force feedback to mobile devices , 2013, CHI.

[50]  Bin Zhang,et al.  Master-Slave Gesture Learning System Based on Functional Electrical Stimulation , 2014, ICIRA.

[51]  Toshiaki Tsuji,et al.  Chattering reduction of functional electrical stimulation with the smith compensator , 2017, IECON 2017 - 43rd Annual Conference of the IEEE Industrial Electronics Society.

[52]  Michael Rohs,et al.  On-skin technologies for muscle sensing and actuation , 2016, UbiComp Adjunct.

[53]  Jun Rekimoto,et al.  Design Guideline for Developing Safe Systems that Apply Electricity to the Human Body , 2018, ACM Trans. Comput. Hum. Interact..

[54]  Michael Rohs,et al.  Zap++: a 20-channel electrical muscle stimulation system for fine-grained wearable force feedback , 2017, MobileHCI.

[55]  Velko Vechev,et al.  DextrES: Wearable Haptic Feedback for Grasping in VR via a Thin Form-Factor Electrostatic Brake , 2018, UIST.

[56]  Makoto Oka,et al.  Feedback Control to Target Joints Angle in Middle Finger PIP and MP Joint Using Functional Electrical Stimulation , 2019, HCI.

[57]  Tomohiko Hayakawa,et al.  Supportive training system for sports skill acquisition based on electrical stimulation , 2017, 2017 IEEE World Haptics Conference (WHC).

[58]  Pedro Lopes,et al.  Affordance++: Allowing Objects to Communicate Dynamic Use , 2015, CHI.

[59]  Cheng Yap Shee,et al.  Using electromechanical delay for real-time anti-phase tremor attenuation system using Functional Electrical Stimulation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[60]  Xiang Li,et al.  FESleeve: A Functional Electrical Stimulation System with Multi-electrode Array for Finger Motion Control , 2016, ICIRA.

[61]  Kenji Suzuki,et al.  A wearable stimulation device for sharing and augmenting kinesthetic feedback , 2015, AH.

[62]  Michael Rohs,et al.  Let me grab this: a comparison of EMS and vibration for haptic feedback in free-hand interaction , 2014, AH.

[63]  Dingguo Zhang,et al.  Wireless Master-Slave FES Rehabilitation System Using sEMG Control , 2012, ICIRA.

[64]  Pedro Lopes,et al.  Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation , 2015, UIST.

[65]  Yoichi Nagashima,et al.  Bio-Sensing Systems and Bio-Feedback Systems for Interactive Media Arts , 2003, NIME.

[66]  M H Schieber,et al.  Quantifying the Independence of Human Finger Movements: Comparisons of Digits, Hands, and Movement Frequencies , 2000, The Journal of Neuroscience.

[67]  Jonna Häkkilä,et al.  EMS Painter: Co-creating Visual Art using Electrical Muscle Stimulation , 2018, TEI.

[68]  Thomas H. Massie,et al.  The PHANToM Haptic Interface: A Device for Probing Virtual Objects , 1994 .

[69]  Takaya Ishimaru,et al.  Virtual bumps display based on electrical muscle stimulation , 2020, 2020 IEEE Haptics Symposium (HAPTICS).

[70]  Tomohiko Hayakawa,et al.  Trajectory adjustment system for learning based on electrical stimulation , 2017, AH.

[71]  Pedro Lopes,et al.  Providing Haptics to Walls & Heavy Objects in Virtual Reality by Means of Electrical Muscle Stimulation , 2017, CHI.

[72]  Kenji Suzuki,et al.  bioSync: A Paired Wearable Device for Blending Kinesthetic Experience , 2017, CHI.

[73]  Jun Rekimoto,et al.  Design and Study of a Multi-Channel Electrical Muscle Stimulation Toolkit for Human Augmentation , 2018, AH.

[74]  Albrecht Schmidt,et al.  Creating user interfaces with electrical muscle stimulation , 2017, Interactions.

[75]  Pedro Lopes,et al.  Muscle-plotter: An Interactive System based on Electrical Muscle Stimulation that Produces Spatial Output , 2016, UIST.

[76]  Yifei Zhang,et al.  Dexmo: An Inexpensive and Lightweight Mechanical Exoskeleton for Motion Capture and Force Feedback in VR , 2016, CHI.

[77]  Shubham Sharma,et al.  Star-Force: A Playful Implementation of the Jedi-force , 2017, Tangible and Embedded Interaction.

[78]  Xin Zhao,et al.  Intelligent algorithm tuning PID method of function electrical stimulation using knee joint angle , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[79]  Pedro Lopes,et al.  Increasing Electrical Muscle Stimulation's Dexterity by means of Back of the Hand Actuation , 2021, CHI.

[80]  Dingguo Zhang,et al.  Influence of volitional contraction on muscle response to functional electrical stimulation , 2014, 2014 IEEE 19th International Functional Electrical Stimulation Society Annual Conference (IFESS).

[81]  Nozomu Hoshimiya,et al.  A method of multichannel PID control of two-degree-of-freedom wrist joint movements by functional electrical stimulation , 2003, Systems and Computers in Japan.

[82]  Pedro Lopes,et al.  Proprioceptive Interaction , 2015, CHI.

[83]  Fan Zhang,et al.  MediaPipe: A Framework for Building Perception Pipelines , 2019, ArXiv.

[84]  J. Blake,et al.  Haptic Glove With MR Brakes for Virtual Reality , 2009, IEEE/ASME Transactions on Mechatronics.

[85]  Hiroyuki Kajimoto,et al.  Optimal selection of electrodes for muscle electrical stimulation using twitching motion measurement , 2013, AH.