Touchsense: classifying finger touches and measuring their force with an electromyography armband

Identifying the finger used for touching and measuring the force of the touch provides valuable information on manual interactions. This information can be inferred from electromyography (EMG) of the forearm, measuring the activation of the muscles controlling the hand and fingers. We present Touch-Sense, which classifies the finger touches using a novel neural network architecture and estimates their force on a smartphone in real time based on data recorded from the sensors of an inexpensive and wireless EMG armband. Using data collected from 18 participants with force ground truth, we evaluate our system's performance and limitations. Our system could allow for new interaction paradigms with appliances and objects, which we exemplarily showcase in four applications.

[1]  Kazunori Okada,et al.  Simple space-domain features for low-resolution sEMG pattern recognition , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[2]  Sebastian Boring,et al.  Designing user-, hand-, and handpart-aware tabletop interactions with the TouchID toolkit , 2011, ITS '11.

[3]  Dana Kulic,et al.  Hand gesture recognition based on surface electromyography , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[4]  Kongqiao Wang,et al.  A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[5]  Karthik Ramani,et al.  Extended multitouch: recovering touch posture and differentiating users using a depth camera , 2012, UIST.

[6]  Daniel Vogel,et al.  Finger-Aware Shortcuts , 2016, CHI.

[7]  Desney S. Tan,et al.  Making muscle-computer interfaces more practical , 2010, CHI.

[8]  Géry Casiez,et al.  WhichFingers: Identifying Fingers on Touch Surfaces and Keyboards using Vibration Sensors , 2017, UIST.

[9]  Antonio Krüger,et al.  EMPress: Practical Hand Gesture Classification with Wrist-Mounted EMG and Pressure Sensing , 2016, CHI.

[10]  Abhishek Ranjan,et al.  Interacting with large displays from a distance with vision-tracked multi-finger gestural input , 2005, SIGGRAPH '06.

[11]  Desney S. Tan,et al.  Enhancing input on and above the interactive surface with muscle sensing , 2009, ITS '09.

[12]  Tanja Schultz,et al.  Advancing Muscle-Computer Interfaces with High-Density Electromyography , 2015, CHI.

[13]  Mohan S. Kankanhalli,et al.  Semi-Supervised Learning for Surface EMG-based Gesture Recognition , 2017, IJCAI.

[14]  Patrick Baudisch,et al.  Fiberio: a touchscreen that senses fingerprints , 2013, UIST.

[15]  Kevin B. Englehart,et al.  A robust, real-time control scheme for multifunction myoelectric control , 2003, IEEE Transactions on Biomedical Engineering.

[16]  Maury A Nussbaum,et al.  Estimation of forces exerted by the fingers using standardised surface electromyography from the forearm , 2008, Ergonomics.

[17]  Erik J. Scheme,et al.  On the robustness of EMG features for pattern recognition based myoelectric control; A multi-dataset comparison , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[18]  Ravin Balakrishnan,et al.  DualKey: Miniature Screen Text Entry via Finger Identification , 2016, CHI.

[19]  Yasuhito Suenaga,et al.  “FingeRing”: a full-time wearable interface , 1994, CHI '94.

[20]  Desney S. Tan,et al.  Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces , 2008, CHI.

[21]  Antonio Krüger,et al.  User-independent real-time hand gesture recognition based on surface electromyography , 2017, MobileHCI.

[22]  Kongqiao Wang,et al.  Hand Gesture Recognition Research Based on Surface EMG Sensors and 2D-accelerometers , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[23]  Jonna Häkkilä,et al.  Exploring finger specific touch screen interaction for mobile phone user interfaces , 2014, OZCHI.

[24]  Yu Hu,et al.  Surface EMG-Based Inter-Session Gesture Recognition Enhanced by Deep Domain Adaptation , 2017, Sensors.

[25]  Hugo Fuks,et al.  Beauty tech nails: interactive technology at your fingertips , 2014, TEI '14.