Sign Language Recognition System using TensorFlow Object Detection API

Communication is defined as the act of sharing or exchanging information, ideas or feelings. To establish communication between two people, both of them are required to have knowledge and understanding of a common language. But in the case of deaf and dumb people, the means of communication are different. Deaf is the inability to hear and dumb is the inability to speak. They communicate using sign language among themselves and with normal people but normal people do not take seriously the importance of sign language. Not everyone possesses the knowledge and understanding of sign language which makes communication difficult between a normal person and a deaf and dumb person. To overcome this barrier, one can build a model based on machine learning. A model can be trained to recognize different gestures of sign language and translate them into English. This will help a lot of people in communicating and conversing with deaf and dumb people. The existing Indian Sing Language Recognition systems are designed using machine learning algorithms with single and double-handed gestures but they are not real-time. In this paper, we propose a method to create an Indian Sign Language dataset using a webcam and then using transfer learning, train a TensorFlow model to create a real-time Sign Language Recognition system. The system achieves a good level of accuracy even with a limited size dataset.

[1]  Changshui Zhang,et al.  A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training , 2019, IEEE Transactions on Multimedia.

[2]  Alan W. C. Tan,et al.  Block-based histogram of optical flow for isolated sign language recognition , 2016, J. Vis. Commun. Image Represent..

[3]  Ying Xie,et al.  American Sign Language Recognition using Deep Learning and Computer Vision , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[4]  Oscar Koller,et al.  Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Parteek Kumar,et al.  Sign Language Recognition Systems: A Decade Systematic Literature Review , 2019, Archives of Computational Methods in Engineering.

[6]  Lihong Zheng,et al.  Recent Advances of Deep Learning for Sign Language Recognition , 2017, 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[7]  P. V. V. Kishore,et al.  4-Camera model for sign language recognition using elliptical fourier descriptors and ANN , 2015, 2015 International Conference on Signal Processing and Communication Engineering Systems.

[8]  Alan W. C. Tan,et al.  A feature covariance matrix with serial particle filter for isolated sign language recognition , 2016, Expert Syst. Appl..

[9]  B. Thilagavathi,et al.  Sign language recognition system , 2015, 2015 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS).

[10]  Meredith Ringel Morris,et al.  Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective , 2019, ASSETS.

[11]  Mohamed A. Deriche,et al.  Arabic sign language recognition using the leap motion controller , 2014, 2014 IEEE 23rd International Symposium on Industrial Electronics (ISIE).

[12]  Mieke Van Herreweghe,et al.  Gesture and Sign Language Recognition with Temporal Residual Networks , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[13]  Nilanjan Dey,et al.  Indian Sign Language Recognition Using Optimized Neural Networks , 2015, ITITS.

[14]  Petros Daras,et al.  SIGN LANGUAGE RECOGNITION BASED ON HAND AND BODY SKELETAL DATA , 2018, 2018 - 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[15]  J. R. MacDonald.  The Government of India , 2017 .

[16]  Zhi-jie Liang,et al.  3D Convolutional Neural Networks for Dynamic Sign Language Recognition , 2018, Comput. J..

[17]  Mohamed-Slim Alouini,et al.  Types of Communication , 2005 .

[18]  Changshui Zhang,et al.  Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Wen Gao,et al.  A Chinese sign language recognition system based on SOFM/SRN/HMM , 2004, Pattern Recognit..

[20]  Zaid Omar,et al.  A review of hand gesture and sign language recognition techniques , 2017, International Journal of Machine Learning and Cybernetics.

[21]  Diego H. Peluffo-Ordóñez,et al.  Sign Language Recognition Based on Intelligent Glove Using Machine Learning Techniques , 2018, 2018 IEEE Third Ecuador Technical Chapters Meeting (ETCM).

[22]  D G Enikeev,et al.  Sign language recognition through Leap Motion controller and input prediction algorithm , 2021 .

[23]  Karen Das,et al.  Indian Sign Language Recognition Using Eigen Value Weighted Euclidean Distance Based Classification Technique , 2013, ArXiv.

[24]  Aarti G. Ambekar,et al.  Sign language recognition using image based hand gesture recognition techniques , 2016, 2016 Online International Conference on Green Engineering and Technologies (IC-GET).

[25]  Xiaoping Li,et al.  IoU-balanced Loss Functions for Single-stage Object Detection , 2019, Pattern Recognit. Lett..

[26]  Siddharth Swarup Rautaray,et al.  A Real Time Hand Tracking System for Interactive Applications , 2011 .

[27]  Ashish Sharma,et al.  Benchmarking deep neural network approaches for Indian Sign Language recognition , 2020, Neural Computing and Applications.

[28]  Jie Huang,et al.  Video-based Sign Language Recognition without Temporal Segmentation , 2018, AAAI.

[29]  Kusumika Krori Dutta,et al.  Machine Learning Techniques for Indian Sign Language Recognition , 2017, 2017 International Conference on Current Trends in Computer, Electrical, Electronics and Communication (CTCEEC).

[30]  Nguyen Duc Dung,et al.  A Comparison of SimpSVM and RVM for Sign Language Recognition , 2017, ICMLSC.

[31]  F. Wong,et al.  Hidden Markov Model-Based Gesture Recognition with Overlapping Hand-Head/Hand-Hand Estimated Using Kalman Filter , 2012, 2012 Third International Conference on Intelligent Systems Modelling and Simulation.

[32]  Fei Huang,et al.  Hand Tracking Algorithm Based on SuperPixels Feature , 2013, 2013 International Conference on Information Science and Cloud Computing Companion.

[33]  D. Tewari,et al.  A Visual Recognition of Static Hand Gestures in Indian Sign Language based on Kohonen Self-Organizing Map Algorithm , 2012 .

[34]  Debi Prosad Dogra,et al.  Independent Bayesian classifier combination based sign language recognition using facial expression , 2018, Inf. Sci..

[35]  Houqiang Li,et al.  Iterative Alignment Network for Continuous Sign Language Recognition , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).