Continuous sign language recognition from wearable IMUs using deep capsule networks and game theory

Abstract Sign Language is used by the deaf community all over world. The work presented here proposes a novel one-dimensional deep capsule network (CapsNet) architecture for continuous Indian Sign Language recognition by means of signals obtained from a custom designed wearable IMU system. The performance of the proposed CapsNet architecture is assessed by altering dynamic routing between capsule layers. The proposed CapsNet yields improved accuracy values of 94% for 3 routings and 92.50% for 5 routings in comparison with the convolutional neural network (CNN) that yields an accuracy of 87.99%. Improved learning of the proposed architecture is also validated by spatial activations depicting excited units at the predictive layer. Finally, a novel non-cooperative pick-and-predict competition is designed between CapsNet and CNN. Higher value of Nash equilibrium for CapsNet as compared to CNN indicates the suitability of the proposed approach.

[1]  Kosin Chamnongthai,et al.  Fist american sign language recognition using leap motion sensor , 2018, 2018 International Workshop on Advanced Image Technology (IWAIT).

[2]  Karush Suri,et al.  Transfer Learning for sEMG-based Hand Gesture Classification using Deep Learning in a Master- Slave Architecture , 2018, 2018 3rd International Conference on Contemporary Computing and Informatics (IC3I).

[3]  Qiang Zhang,et al.  Classification of ECG signals based on 1D convolution neural network , 2017, 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom).

[4]  M. Mahesh,et al.  Sign language translator for mobile platforms , 2017, 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI).

[5]  Karush Suri,et al.  Classification of Hand Gestures from Wearable IMUs using Deep Neural Network , 2018, 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT).

[6]  Yingnan Sun,et al.  An Artificial Neural Network Framework for Gait-Based Biometrics , 2019, IEEE Journal of Biomedical and Health Informatics.

[7]  Masayoshi Tomizuka,et al.  A sEMG Classification Framework with Less Training Data , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[8]  Rong Zhang,et al.  A Three-Stage Accelerometer Self-Calibration Technique for Space-Stable Inertial Navigation Systems , 2018, Sensors.

[9]  Aleksander Madry,et al.  A Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations , 2017, ArXiv.

[10]  Dale Schuurmans,et al.  Deep Learning Games , 2016, NIPS.

[11]  Hung Manh La,et al.  A human foot motion localization algorithm using IMU , 2016, 2016 American Control Conference (ACC).

[12]  Liang Shi,et al.  Motion Control in VR — Real-time Upper Limb Tracking via IMU and Flex Sensor , 2016 .

[13]  Yi Chen,et al.  Error Contributions during MEMS Gyroscope Calibration by Chip-Scale Micro-Stage with Capacitive Motion Sensor , 2018, 2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM).

[14]  Li Chen,et al.  Image recognition based on deep learning , 2015, 2015 Chinese Automation Congress (CAC).

[15]  Björn W. Schuller,et al.  LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework , 2013, Image Vis. Comput..

[16]  William Tomkins,et al.  Indian Sign Language , 1969 .

[17]  Debdas Ghosh,et al.  Nash equilibrium strategy for non-cooperative games with interval type-2 fuzzy payoffs , 2018, 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE).

[18]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[19]  Angkoon Phinyomark,et al.  Feature Extraction and Selection for Myoelectric Control Based on Wearable EMG Sensors , 2018, Sensors.

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[22]  Chen Xu,et al.  MS-CapsNet: A Novel Multi-Scale Capsule Network , 2018, IEEE Signal Processing Letters.

[23]  Theodore Antonakopoulos,et al.  An IMU-Based Wearable System for Automatic Pointing During Presentations , 2016 .

[24]  Sercan Sunetci,et al.  Semantic image segmentation with deep features , 2018, 2018 26th Signal Processing and Communications Applications Conference (SIU).

[25]  Roozbeh Jafari,et al.  A Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors , 2016, IEEE Journal of Biomedical and Health Informatics.

[26]  Jafar A. Alzubi,et al.  Optimal Classifier Ensemble Design Based on Cooperative Game Theory , 2015 .

[27]  Edwin D. de Jong,et al.  GANGs: Generative Adversarial Network Games , 2017, ArXiv.