Performance analysis of Otsu thresholding for sign language segmentation

Sign language recognition system generally consists of three main processes, which are segmentation, modelling, and classification. Image segmentation plays a crucial role as the initial step in sign language recognition. Despite the many sign language recognition system algorithms proposed in the literature and their well-understood usage, their performance analyses are relatively limited. As such, the main motivation of this paper is to critically analyse the feasibility of successful sign language segmentation under variation of dynamic scene parameters such as noise, hand size, and intensity difference between hand and background. The focus is on image thresholding using Otsu technique, since it is the most commonly used in initial process of sign language segmentation. The analysis of this work was developed based on Monte Carlo statistical method, which showed that the success of sign language segmentation depends on hand size, hand background intensity difference, and noise measurement. The result showed that the sign alphabets with handheld shape like A, E, I, M, N, S, and T is easier to segment, while sign alphabets with finger-extend shape like C, D, F, G, H, K, L, P, R, U, V, W, and Y is harder to segment. Experiment using real images demonstrate the capability of the conditions to correctly predict the outcome of sign language segmentation using Otsu technique. In conclusion, the success of sign language segmentation could be predicted beforehand with obtainable scene parameters.

[1]  A. Sharmila Konwar,et al.  An American Sign Language detection system using HSV color model and edge detection , 2014, 2014 International Conference on Communication and Signal Processing.

[2]  Stan Sclaroff,et al.  Sign Language Spotting with a Threshold Model Based on Conditional Random Fields , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Alireza Bab-Hadiashar,et al.  Analysis of planar-motion segmentation using affine fundamental matrix , 2014, IET Comput. Vis..

[4]  Haniza Yazid,et al.  Performance analysis of image thresholding: Otsu technique , 2018 .

[5]  Federico Sandoval-Ibarra,et al.  American Sign Language Alphabet Recognition Using a Neuromorphic Sensor and an Artificial Neural Network , 2017, Sensors.

[6]  Qian Zhang,et al.  Research of improving semantic image segmentation based on a feature fusion model , 2020, Journal of Ambient Intelligence and Humanized Computing.

[7]  Debi Prosad Dogra,et al.  Coupled HMM-based multi-sensor data fusion for sign language recognition , 2017, Pattern Recognit. Lett..

[8]  Fan Chen,et al.  Hand Gesture Detection and Segmentation Based on Difference Background Image with Complex Background , 2008, 2008 International Conference on Embedded Software and Systems.

[9]  Nicolas Pugeault,et al.  Spelling it out: Real-time ASL fingerspelling recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[10]  Asha Thalange,et al.  COHST and Wavelet Features Based Static ASL Numbers Recognition , 2016 .

[11]  Ahmad Zaki Shukor,et al.  A New Data Glove Approach for Malaysian Sign Language Detection , 2015 .

[12]  Harsh Shah,et al.  Survey on Vision based Hand Gesture Recognition , 2019 .

[13]  Kumud Tripathi,et al.  Continuous Indian Sign Language Gesture Recognition and Sentence Formation , 2015 .

[14]  S. Basah,et al.  Conditions for motion-background segmentation using fundamental matrix , 2009, DICTA 2009.

[15]  Joanna Isabelle Olszewska,et al.  British Sign Language Recognition In The Wild Based On Multi-Class SVM , 2019, 2019 Federated Conference on Computer Science and Information Systems (FedCSIS).

[16]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[17]  Heidy Sierra,et al.  American sign language translation using edge detection and cross correlation , 2017, 2017 IEEE Colombian Conference on Communications and Computing (COLCOM).

[18]  Nayan M. Kakoty,et al.  Recognition of Sign Language Alphabets and Numbers based on Hand Kinematics using A Data Glove , 2018 .

[19]  Zaid Omar,et al.  A review of hand gesture and sign language recognition techniques , 2017, International Journal of Machine Learning and Cybernetics.

[20]  Adam Schembri,et al.  Australian Sign Language: Auslan: An Introduction to Sign Language Linguistics , 2007 .

[21]  Nitin V. Pujari,et al.  Finger Detection for Sign Language Recognition , 2009 .

[22]  S. Ogden,et al.  A G protein functions immediately downstream of Smoothened in Hedgehog signaling , 2008, Nature.

[23]  Alireza Bab-Hadiashar,et al.  Limits of Motion-Background Segmentation Using Fundamental Matrix Estimation , 2008, 2008 Digital Image Computing: Techniques and Applications.

[24]  Hee-Deok Yang,et al.  Sign Language Recognition with the Kinect Sensor Based on Conditional Random Fields , 2014, Sensors.

[25]  Liu Hong-zhe,et al.  GrabCut image segmentation algorithm based on structure tensor , 2017 .

[26]  Jasni Mohamad Zain,et al.  Application of Threshold Techniques for Readability Improvement of Jawi Historical Manuscript Images , 2011, ArXiv.

[27]  Haitham Badi.  Recent methods in vision-based hand gesture recognition , 2016, International Journal of Data Science and Analytics.

[28]  David Suter,et al.  Finite Sample Bias of Robust Estimators in Segmentation of Closely Spaced Structures: A Comparative Study , 2010, Journal of Mathematical Imaging and Vision.

[29]  Shan Lu,et al.  Color-based hands tracking system for sign language recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[30]  Gabriel Sanchez-Perez,et al.  Shape-based hand recognition approach using the morphological pattern spectrum , 2009, J. Electronic Imaging.

[31]  Benjamin Schrauwen,et al.  Sign Language Recognition Using Convolutional Neural Networks , 2014, ECCV Workshops.

[32]  Muñoz A. Jaime,et al.  Raise Awareness in Society About Deafness: A Proposal with Learning Objects and Scenarios☆ , 2016 .

[33]  Khaled Assaleh,et al.  Glove-Based Continuous Arabic Sign Language Recognition in User-Dependent Mode , 2015, IEEE Transactions on Human-Machine Systems.

[34]  Alireza Bab-Hadiashar,et al.  Conditions for Segmentation of 2D Translations of 3D Objects , 2009, ICIAP.

[35]  Zhiyu Qin,et al.  A kind of method for selection of optimum threshold for segmentation of digital color plane image , 2008, 2008 9th International Conference on Computer-Aided Industrial Design and Conceptual Design.

[36]  Astha Baxi,et al.  A Review on Otsu Image Segmentation Algorithm , 2013 .

[37]  Er. Rajandeep Kaur,et al.  Review of Image Segmentation Technique , 2017 .

[38]  Sung Kwan Kang,et al.  Color Based Hand and Finger Detection Technology for User Interaction , 2008, 2008 International Conference on Convergence and Hybrid Information Technology.

[39]  Wenjin Tao,et al.  American Sign Language alphabet recognition using Convolutional Neural Networks with multiview augmentation and inference fusion , 2018, Eng. Appl. Artif. Intell..

[40]  Mazen M. Selim,et al.  An Automatic Arabic Sign Language Recognition System (ArSLRS) , 2017, J. King Saud Univ. Comput. Inf. Sci..

[41]  Zhongfu Ye,et al.  Continuous sign language recognition using level building based on fast hidden Markov model , 2016, Pattern Recognit. Lett..

[42]  Muhammad Modi bin Lakulu,et al.  A Review on Systems-Based Sensory Gloves for Sign Language Recognition State of the Art between 2007 and 2017 , 2018, Sensors.

[43]  Manoochehr Nahvi,et al.  An algorithm on sign words extraction and recognition of continuous Persian sign language based on motion and shape features of hands , 2018, Pattern Analysis and Applications.