Trajectory-Based Recognition of Dynamic Persian Sign Language Using Hidden Markov Model

Abstract Sign Language Recognition (SLR) is an important step in facilitating the communication among deaf people and the rest of society. Existing Persian sign language recognition systems are mainly restricted to static signs which are not very useful in everyday communications. In this study, a dynamic Persian sign language recognition system is presented. A collection of 1200 videos were captured from 12 individuals performing 20 dynamic signs with a simple white glove. The trajectory of the hands, along with hand shape information were extracted from each video using a simple region-growing technique. These time-varying trajectories were then modeled using Hidden Markov Model (HMM) with Gaussian probability density functions as observations. The performance of the system was evaluated in different experimental strategies. Signer-independent and signer-dependent experiments were performed on the proposed system and the average accuracy of 97.48% was obtained. The experimental results demonstrated that the performance of the system is independent of the subject and it can also perform excellently even with a limited number of training data.

[1]  Dimitris N. Metaxas,et al.  Parallel hidden Markov models for American sign language recognition , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[2]  Ali Asghar Zare,et al.  Recognition of a real-time signer-independent static Farsi sign language based on fourier coefficients amplitude , 2018, Int. J. Mach. Learn. Cybern..

[3]  Zaid Omar,et al.  A review of hand gesture and sign language recognition techniques , 2017, International Journal of Machine Learning and Cybernetics.

[4]  Milad Moghaddam,et al.  Static Persian Sign Language Recognition Using Kernel-Based Feature Extraction , 2011, 2011 7th Iranian Conference on Machine Vision and Image Processing.

[5]  Moritz Knorr,et al.  The significance of facial features for automatic sign language recognition , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[6]  Roozbeh Jafari,et al.  Real-time American Sign Language Recognition using wrist-worn motion and surface EMG sensors , 2015, 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[7]  Khaled Assaleh,et al.  Video-based signer-independent Arabic sign language recognition using hidden Markov models , 2009, Appl. Soft Comput..

[8]  Wen Gao,et al.  Large-Vocabulary Continuous Sign Language Recognition Based on Transition-Movement Models , 2007, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[9]  Alaa Barkoky,et al.  Static hand gesture recognition of Persian sign numbers using thinning method , 2011, 2011 International Conference on Multimedia Technology.

[10]  Chung-Lin Huang,et al.  Hand gesture recognition using a real-time tracking method and hidden Markov models , 2003, Image Vis. Comput..

[11]  Hermann Ney,et al.  Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers , 2015, Comput. Vis. Image Underst..

[12]  M. Deriche,et al.  A signer-independent Arabic Sign Language recognition system using face detection, geometric features, and a Hidden Markov Model , 2012, Comput. Electr. Eng..

[13]  Sakshi Lahoti,et al.  Android Based American Sign Language Recognition System with Skin Segmentation and SVM , 2018, 2018 9th International Conference on Computing, Communication and Networking Technologies (ICCCNT).

[14]  Barbara Rita Barricelli,et al.  A visual language and interactive system for end-user development of internet of things ecosystems , 2017, J. Vis. Lang. Comput..

[15]  Ali Karami,et al.  Persian sign language (PSL) recognition using wavelet transform and neural networks , 2011, Expert Syst. Appl..

[16]  Nilanjan Dey,et al.  Indian Sign Language Recognition Using Optimized Neural Networks , 2015, ITITS.

[17]  Hermann Ney,et al.  Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition , 2016, BMVC.

[18]  Hermann Ney,et al.  Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Wen Gao,et al.  Sign Language Recognition Based on HMM/ANN/DP , 2000, Int. J. Pattern Recognit. Artif. Intell..

[20]  Hermann Ney,et al.  Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Hamid Amiri,et al.  Hand Gesture Recognition Using Leap Motion Controller for Recognition of Arabic Sign Language , 2016 .

[22]  M. Maraqa,et al.  Recognition of Arabic Sign Language (ArSL) using recurrent neural networks , 2008, 2008 First International Conference on the Applications of Digital Information and Web Technologies (ICADIWT).

[23]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[24]  Debi Prosad Dogra,et al.  Coupled HMM-based multi-sensor data fusion for sign language recognition , 2017, Pattern Recognit. Lett..

[25]  Alex Pentland,et al.  Coupled hidden Markov models for complex action recognition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[26]  Khaled Assaleh,et al.  Glove-Based Continuous Arabic Sign Language Recognition in User-Dependent Mode , 2015, IEEE Transactions on Human-Machine Systems.

[27]  Hermann Ney,et al.  Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs , 2018, International Journal of Computer Vision.

[28]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[29]  Tamer Shanableh,et al.  Spatio-Temporal Feature-Extraction Techniques for Isolated Gesture Recognition in Arabic Sign Language , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[30]  Mohamed A. Deriche,et al.  Image based arabic sign language recognition , 2005, Proceedings of the Eighth International Symposium on Signal Processing and Its Applications, 2005..

[31]  B. Parton.  Sign language recognition and translation: a multidisciplined approach from the field of artificial intelligence. , 2005, Journal of deaf studies and deaf education.

[32]  Robyn A. Owens,et al.  Australian sign language recognition , 2005, Machine Vision and Applications.

[33]  Dan Schonfeld,et al.  Object Trajectory-Based Activity Classification and Recognition Using Hidden Markov Models , 2007, IEEE Transactions on Image Processing.

[34]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[35]  Alan W. C. Tan,et al.  A feature covariance matrix with serial particle filter for isolated sign language recognition , 2016, Expert Syst. Appl..

[36]  Hermann Ney,et al.  Deep Learning of Mouth Shapes for Sign Language , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[37]  Zhongfu Ye,et al.  A Novel Chinese Sign Language Recognition Method Based on Keyframe-Centered Clips , 2018, IEEE Signal Processing Letters.

[38]  José Colás Pasamontes,et al.  Spanish Sign Language synthesis system , 2012, J. Vis. Lang. Comput..