HMM-based Arabic Sign Language Recognition using Kinect

With research in Arabic Sign Language Recognition (ArSLR) still in its infancy, we present a method for recognition of Arabic Sign Language (ArSL) using Microsoft Kinect. A new dataset for ArSL words was collected using Kinect due to the lack of existence of one. With the intention of use in medical hospitals to aid communication between a deaf or hard-of-hearing patient with the doctor, the foremost goal was presenting a robust system, which does not impose any constraints on either the signer or the background. The proposed system combines skeletal data and depth information for hand tracking and segmentation, without relying on any color markers, or skin color detection algorithms. The extracted features describe the four elements of the hand that are used to describe the phonological structure of ArSL: articulation point, hand orientation, hand shape, and hand movement. Hidden Markov Model (HMM) was used for classification using ten-fold cross-validation, achieving an accuracy of 80.47%. Singer-independent experiments resulted in an average recognition accuracy of 64.61%.

[1]  Mohamed A. Ismail,et al.  New algorithm for matching 2D objects , 2002, IS&T/SPIE Electronic Imaging.

[2]  Hee-Deok Yang,et al.  Sign Language Recognition with the Kinect Sensor Based on Conditional Random Fields , 2014, Sensors.

[3]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[4]  Nooritawati Md Tahir,et al.  Review in Sign Language Recognition Systems , 2012, 2012 IEEE Symposium on Computers & Informatics (ISCI).

[5]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[6]  Mohamed Jemni,et al.  Toward the creation of an Arab Gloss for arabic Sign Language annotation , 2013, Fourth International Conference on Information and Communication Technology and Accessibility (ICTA).

[7]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Mohamed A. Ismail,et al.  Matching Occluded Objects Invariant to Rotations, Translations, Reflections, and Scale Changes , 2003, SCIA.

[9]  Thad Starner,et al.  American sign language recognition with the kinect , 2011, ICMI '11.

[10]  Mariusz Oszust,et al.  Polish sign language words recognition with Kinect , 2013, 2013 6th International Conference on Human System Interactions (HSI).

[11]  Yasser El-Sonbaty,et al.  Compressing Sets of Similar Medical Images Using Multilevel Centroid Technique , 2003, DICTA.

[12]  Z. Zenn Bien,et al.  A dynamic gesture recognition system for the Korean sign language (KSL) , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[13]  M. Mohandes,et al.  Image-Based and Sensor-Based Approaches to Arabic Sign Language Recognition , 2014, IEEE Transactions on Human-Machine Systems.

[14]  M. Deriche,et al.  A signer-independent Arabic Sign Language recognition system using face detection, geometric features, and a Hidden Markov Model , 2012, Comput. Electr. Eng..

[15]  Luiz Velho,et al.  Kinect and RGBD Images: Challenges and Applications , 2012, 2012 25th SIBGRAPI Conference on Graphics, Patterns and Images Tutorials.

[16]  Xia Liu,et al.  Sign recognition using depth image streams , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[17]  Robyn A. Owens,et al.  Automatic Recognition of Colloquial Australian Sign Language , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[18]  Fabrice Mériaudeau.  Sign Language Translator using Microsoft Kinect , 2012 .

[19]  Jonas Beskow,et al.  Visual Recognition of Isolated Swedish Sign Language Signs , 2012, ArXiv.

[20]  Jason Jianjun Gu,et al.  Combining features for Chinese sign language recognition with Kinect , 2014, 11th IEEE International Conference on Control & Automation (ICCA).

[21]  M. Younus Javed,et al.  A Survey on Sign Language Recognition , 2011, 2011 Frontiers of Information Technology.

[22]  Frederico G. Guimarães,et al.  Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors , 2014, Expert Syst. Appl..

[23]  Hermann Hienz,et al.  Relevant features for video-based continuous sign language recognition , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).