Deep Learning-Based Approach for Sign Language Gesture Recognition With Efficient Hand Gesture Representation

Hand gesture recognition is an attractive research field with a wide range of applications, including video games and telesurgery techniques. Another important application of hand gesture recognition is the translation of sign language, which is a complicated structured form of hand gestures. In sign language, the fingers’ configuration, the hand’s orientation, and the hand’s relative position to the body are the primitives of structured expressions. The importance of hand gesture recognition has increased due to the prevalence of touchless applications and the rapid growth of the hearing-impaired population. However, developing an efficient recognition system needs to overcome the challenges of hand segmentation, local hand shape representation, global body configuration representation, and gesture sequence modeling. In this paper, a novel system is proposed for dynamic hand gesture recognition using multiple deep learning architectures for hand segmentation, local and global feature representations, and sequence feature globalization and recognition. The proposed system is evaluated on a very challenging dataset, which consists of 40 dynamic hand gestures performed by 40 subjects in an uncontrolled environment. The results show that the proposed system outperforms state-of-the-art approaches, demonstrating its effectiveness.

[1]  Benjamin Schrauwen,et al.  Sign Language Recognition Using Convolutional Neural Networks , 2014, ECCV Workshops.

[2]  Houqiang Li,et al.  Sign Language Recognition Based on Trajectory Modeling with HMMs , 2016, MMM.

[3]  Emil M. Petriu,et al.  Dynamic Sign Language Recognition for Smart Home Interactive Application Using Stochastic Linear Formal Grammar , 2015, IEEE Transactions on Instrumentation and Measurement.

[4]  Pavlo Molchanov,et al.  Hand gesture recognition with 3D convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[5]  Lorenzo Torresani,et al.  Learning Spatiotemporal Features with 3D Convolutional Networks , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Luis A. Guerrero,et al.  Improving Deaf People Accessibility and Communication Through Automatic Sign Language Recognition Using Novel Technologies , 2016 .

[7]  Rajiv Ranjan Sahay,et al.  Deep Gesture: Static Hand Gesture Recognition Using CNN , 2016, CVIP.

[8]  Saleh Aly,et al.  Arabic Sign Language Recognition Using Spatio-Temporal Local Binary Patterns and Support Vector Machine , 2014, AMLTA.

[9]  Alan W. C. Tan,et al.  A feature covariance matrix with serial particle filter for isolated sign language recognition , 2016, Expert Syst. Appl..

[10]  M. Younus Javed,et al.  A Survey on Sign Language Recognition , 2011, 2011 Frontiers of Information Technology.

[11]  Cordelia Schmid,et al.  Long-Term Temporal Convolutions for Action Recognition , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Tzuu-Hseng S. Li,et al.  Recognition System for Home-Service-Related Sign Language Using Entropy-Based $K$ -Means Algorithm and ABC-Based HMM , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[13]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Daniel Kelly,et al.  Continuous recognition of motion based gestures in sign language , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[15]  Ke Gao,et al.  DenseImage Network: Video Spatial-Temporal Evolution Encoding and Understanding , 2018, ArXiv.

[16]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Musaed Alhussein,et al.  Voice Pathology Detection Using Deep Learning on Mobile Healthcare Framework , 2018, IEEE Access.

[18]  Daniel Kelly,et al.  A person independent system for recognition of hand postures used in sign language , 2010, Pattern Recognit. Lett..

[19]  Kouichi Murakami,et al.  Gesture recognition using recurrent neural networks , 1991, CHI.

[20]  M. B. Waldron,et al.  Isolated ASL sign recognition system for deaf persons , 1995 .

[21]  Gerhard Rigoll,et al.  Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Muhammad Ghulam,et al.  Hand Gesture Recognition Using 3D-CNN Model , 2020, IEEE Consumer Electronics Magazine.

[23]  T.D. Bui,et al.  Recognizing Postures in Vietnamese Sign Language With MEMS Accelerometers , 2007, IEEE Sensors Journal.

[24]  Ghulam Muhammad,et al.  Hand Gesture Recognition for Sign Language Using 3DCNN , 2020, IEEE Access.

[25]  Saleh Aly,et al.  DeepArSLR: A Novel Signer-Independent Deep Learning Framework for Isolated Arabic Sign Language Gestures Recognition , 2020, IEEE Access.

[26]  M. Işcan,et al.  Estimation of stature from body parts. , 2003, Forensic science international.

[27]  Muhammad Ghulam,et al.  Computing and Processing on the Edge: Smart Pathology Detection for Connected Healthcare , 2019, IEEE Network.

[28]  Bin Hu,et al.  Deep Learning Based Hand Gesture Recognition and UAV Flight Controls , 2018, International Journal of Automation and Computing.

[29]  Ao Tang,et al.  A Real-Time Hand Posture Recognition System Using Deep Neural Networks , 2015, ACM Trans. Intell. Syst. Technol..

[30]  Kongqiao Wang,et al.  A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[31]  Sander Dieleman,et al.  Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video , 2015, International Journal of Computer Vision.

[32]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Jakkree Srinonchat,et al.  Hand Gesture Recognition for Thai Sign Language in Complex Background Using Fusion of Depth and Color Video , 2016 .

[34]  Clément Gosselin,et al.  Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[35]  Vasiliki Kosmidou,et al.  Sign Language Recognition Using Intrinsic-Mode Sample Entropy on sEMG and Accelerometer Data , 2009, IEEE Transactions on Biomedical Engineering.

[36]  Muhammad Ghulam,et al.  Medical Image Forgery Detection for Smart Healthcare , 2018, IEEE Communications Magazine.

[37]  Xilin Chen,et al.  Sparse Observation (SO) Alignment for Sign Language Recognition , 2016, Neurocomputing.

[38]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[39]  Samir I. Shaheen,et al.  Sign language recognition using a combination of new vision based features , 2011, Pattern Recognit. Lett..

[40]  Wen Gao,et al.  Large-Vocabulary Continuous Sign Language Recognition Based on Transition-Movement Models , 2007, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[41]  Jianyu Yang,et al.  Parsing 3D motion trajectory for gesture recognition , 2016, J. Vis. Commun. Image Represent..

[42]  Mubarak Shah,et al.  An End-to-end 3D Convolutional Neural Network for Action Detection and Segmentation in Videos , 2017, ArXiv.