Hand Gesture Recognition Using EGaIn-Silicone Soft Sensors

Exploiting hand gestures for non-verbal communication has extraordinary potential in HCI. A data glove is an apparatus widely used to recognize hand gestures. To improve the functionality of the data glove, a highly stretchable and reliable signal-to-noise ratio sensor is indispensable. To do this, the study focused on the development of soft silicone microchannel sensors using a Eutectic Gallium-Indium (EGaIn) liquid metal alloy and a hand gesture recognition system via the proposed data glove using the soft sensor. The EGaIn-silicone sensor was uniquely designed to include two sensing channels to monitor the finger joint movements and to facilitate the EGaIn alloy injection into the meander-type microchannels. We recruited 15 participants to collect hand gesture dataset investigating 12 static hand gestures. The dataset was exploited to estimate the performance of the proposed data glove in hand gesture recognition. Additionally, six traditional classification algorithms were studied. From the results, a random forest shows the highest classification accuracy of 97.3% and a linear discriminant analysis shows the lowest accuracy of 87.4%. The non-linearity of the proposed sensor deteriorated the accuracy of LDA, however, the other classifiers adequately overcame it and performed high accuracies (>90%).

[1]  Osman Hasan,et al.  Wearable technologies for hand joints monitoring for rehabilitation: A survey , 2018, Microelectron. J..

[2]  Philipp Scholl,et al.  Real-Time and Embedded Detection of Hand Gestures with an IMU-Based Glove , 2018, Informatics.

[3]  Ahmad Zaki Shukor,et al.  A New Data Glove Approach for Malaysian Sign Language Detection , 2015 .

[4]  Alessandro Tognetti,et al.  A Multi-Modal Sensing Glove for Human Manual-Interaction Studies , 2016 .

[5]  Maria Letizia Corradini,et al.  Recognition and Classification of Dynamic Hand Gestures by a Wearable Data-Glove , 2020, SN Computer Science.

[6]  Marco E. Benalcázar,et al.  Real-Time Hand Gesture Recognition Using Surface Electromyography and Machine Learning: A Systematic Literature Review , 2020, Sensors.

[7]  Zaid Omar,et al.  A review of hand gesture and sign language recognition techniques , 2017, International Journal of Machine Learning and Cybernetics.

[8]  Jiacun Wang,et al.  Deep Learning Based Hand Gesture Recognition and UAV Flight Controls , 2020, Int. J. Autom. Comput..

[9]  Onur Avci,et al.  1D Convolutional Neural Networks and Applications: A Survey , 2019, Mechanical Systems and Signal Processing.

[10]  Hang Su,et al.  Depth vision guided hand gesture recognition using electromyographic signals , 2020, Adv. Robotics.

[11]  Reza Langari,et al.  EMG and IMU based real-time HCI using dynamic hand gestures for a multiple-DoF robot arm , 2018, J. Intell. Fuzzy Syst..

[12]  Robert J. Wood,et al.  Influence of cross-sectional geometry on the sensitivity and hysteresis of liquid-phase electronic pressure sensors , 2012 .

[13]  Yongji Wang,et al.  RLSESN-based PID adaptive control for a novel wearable rehabilitation robotic hand driven by PM-TS actuators , 2012, Int. J. Intell. Comput. Cybern..

[14]  Carmel Majidi,et al.  Enhanced performance of microfluidic soft pressure sensors with embedded solid microspheres , 2016 .

[15]  Yan Fu,et al.  A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods , 2020, Sensors.

[16]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties , 1989 .

[17]  Fuchun Sun,et al.  Dynamic Gesture Recognition Using Inertial Sensors-based Data Gloves , 2019, 2019 IEEE 4th International Conference on Advanced Robotics and Mechatronics (ICARM).

[18]  Huosheng Hu,et al.  Support Vector Machine-Based Classification Scheme for Myoelectric Control Applied to Upper Limb , 2008, IEEE Transactions on Biomedical Engineering.

[19]  Stéphanie P. Lacour,et al.  Design and functional evaluation of an epidermal strain sensing system for hand tracking , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[20]  Giovanni Saggio,et al.  A novel array of flex sensors for a goniometric glove , 2014 .

[21]  Xiaomin Ren,et al.  Tracing the Motion of Finger Joints for Gesture Recognition via Sewing RGO-Coated Fibers Onto a Textile Glove , 2019, IEEE Sensors Journal.

[22]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[23]  Edoardo Battaglia,et al.  A Synergy-Based Optimally Designed Sensing Glove for Functional Grasp Recognition , 2016, Sensors.

[24]  Joo Chuan Yeo,et al.  Flexible and Stretchable Strain Sensing Actuator for Wearable Soft Robotic Applications , 2016 .

[25]  Robert J. Wood,et al.  Biocompatible Sensors: Biocompatible Soft Fluidic Strain and Force Sensors for Wearable Devices (Adv. Funct. Mater. 7/2019) , 2019, Advanced Functional Materials.

[26]  Sangki Park,et al.  A knitted glove sensing system with compression strain for finger movements , 2018 .

[27]  Robert J. Wood,et al.  Stretchable circuits and sensors for robotic origami , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[28]  Robert J. Wood,et al.  Biocompatible Soft Fluidic Strain and Force Sensors for Wearable Devices , 2018, Advanced functional materials.

[29]  Kyeong-Beom Park,et al.  User Interactions for Augmented Reality Smart Glasses: A Comparative Evaluation of Visual Contexts and Interaction Gestures , 2019, Applied Sciences.

[30]  Rodina Binti Ahmad,et al.  A systematic literature review on vision based gesture recognition techniques , 2018, Multimedia Tools and Applications.

[31]  Olga Sorkine-Hornung,et al.  Deformation Capture via Soft and Stretchable Sensor Arrays , 2018, ACM Trans. Graph..

[32]  Rebecca K. Kramer,et al.  Direct Writing of Gallium‐Indium Alloy for Stretchable Electronics , 2014 .

[33]  Ashok Kumar Sahoo,et al.  SIGN LANGUAGE RECOGNITION: STATE OF THE ART , 2014 .

[34]  Mehmet Akif Ozdemir,et al.  EMG based Hand Gesture Classification using Empirical Mode Decomposition Time-Series and Deep Learning , 2020, 2020 Medical Technologies Congress (TIPTEKNO).

[35]  John Nassour,et al.  A Robust Data-Driven Soft Sensory Glove for Human Hand Motions Identification and Replication , 2020, IEEE Sensors Journal.

[36]  Shwetak N. Patel,et al.  Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing , 2016, CHI.

[37]  A. Lemasson,et al.  The origins of gestures and language: history, current advances and proposed theories , 2019, Biological reviews of the Cambridge Philosophical Society.

[38]  Filbert H. Juwono,et al.  Multi-Features Capacitive Hand Gesture Recognition Sensor: A Machine Learning Approach , 2021, IEEE Sensors Journal.

[39]  Zheng Wang,et al.  A soft stretchable bending sensor and data glove applications , 2016, 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR).

[40]  Suin Kim,et al.  A Novel All-in-One Manufacturing Process for a Soft Sensor System and its Application to a Soft Sensing Glove , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[41]  Yong-Lae Park,et al.  Design and Fabrication of Soft Artificial Skin Using Embedded Microchannels and Liquid Conductors , 2012, IEEE Sensors Journal.

[42]  G. Whitesides,et al.  Eutectic Gallium‐Indium (EGaIn): A Liquid Metal Alloy for the Formation of Stable Structures in Microchannels at Room Temperature , 2008 .

[43]  Giovanni Saggio,et al.  Sign Language Recognition Using Wearable Electronics: Implementing k-Nearest Neighbors with Dynamic Time Warping and Convolutional Neural Network Algorithms , 2020, Sensors.

[44]  Yiwei Tao,et al.  Wearable soft artificial skin for hand motion detection with embedded microfluidic strain sensing , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[45]  Suin Kim,et al.  A Soft Sensor-Based Three-Dimensional (3-D) Finger Motion Measurement System , 2017, Sensors.

[46]  Jia-Jia Li,et al.  Wearable Wide-Range Strain Sensors Based on Ionic Liquids and Monitoring of Human Activities , 2017, Sensors.

[47]  Anupam Agrawal,et al.  Vision based hand gesture recognition for human computer interaction: a survey , 2012, Artificial Intelligence Review.

[48]  Hanchao Yu,et al.  Learning Effective Spatial–Temporal Features for sEMG Armband-Based Gesture Recognition , 2020, IEEE Internet of Things Journal.

[49]  Hyung Jin Sung,et al.  Highly Stretchable, Hysteresis-Free Ionic Liquid-Based Strain Sensor for Precise Human Motion Monitoring. , 2017, ACS applied materials & interfaces.

[50]  Bor-Shing Lin,et al.  Data glove embedded with 9-axis IMU and force sensing sensors for evaluation of hand function , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[51]  Daniel M. Vogt,et al.  Batch Fabrication of Customizable Silicone‐Textile Composite Capacitive Strain Sensors for Human Motion Tracking , 2017 .

[52]  Harry Zhang,et al.  Exploring Conditions For The Optimality Of Naïve Bayes , 2005, Int. J. Pattern Recognit. Artif. Intell..

[53]  M. Dickey.  Stretchable and Soft Electronics using Liquid Metals , 2017, Advanced materials.

[54]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[55]  Muhammad Modi bin Lakulu,et al.  A Review on Systems-Based Sensory Gloves for Sign Language Recognition State of the Art between 2007 and 2017 , 2018, Sensors.

[56]  Yeh-Liang Hsu,et al.  A Review of Accelerometry-Based Wearable Motion Detectors for Physical Activity Monitoring , 2010, Sensors.

[57]  Yu Peng,et al.  Development and evaluation of a sensor glove for hand function assessment and preliminary attempts at assessing hand coordination , 2016 .

[58]  Ying Huang,et al.  Static and Dynamic Human Arm/Hand Gesture Capturing and Recognition via Multiinformation Fusion of Flexible Strain Sensors , 2020, IEEE Sensors Journal.