A Novel String Grammar Unsupervised Possibilistic C-Medians Algorithm for Sign Language Translation Systems

Sign language is a basic method for solving communication problems between deaf and hearing people. In order to communicate, deaf and hearing people normally use hand gestures, which include a combination of hand positioning, hand shapes, and hand movements. Thai Sign Language is the communication method for Thai hearing-impaired people. Our objective is to improve the dynamic Thai Sign Language translation method with a video captioning technique that does not require prior hand region detection and segmentation through using the Scale Invariant Feature Transform (SIFT) method and the String Grammar Unsupervised Possibilistic C-Medians (sgUPCMed) algorithm. This work is the first to propose the sgUPCMed algorithm to cope with the unsupervised generation of multiple prototypes in the possibilistic sense for string data. In our experiments, the Thai Sign Language data set (10 isolated sign language words) was collected from 25 subjects. The best average result within the constrained environment of the blind test data sets of signer-dependent cases was 89–91%, and the successful rate of signer semi-independent cases was 81–85%, on average. For the blind test data sets of signer-independent cases, the best average classification rate was 77–80%. The average result of the system without a constrained environment was around 62–80% for the signer-independent experiments. To show that the proposed algorithm can be implemented in other sign languages, the American sign language (RWTH-BOSTON-50) data set, which consists of 31 isolated American Sign Language words, is also used in the experiment. The system provides 88.56% and 91.35% results on the validation set alone, and for both the training and validation sets, respectively.

[1]  Alfons Juan-Císcar,et al.  On the use of normalized edit distances and an efficient k-NN search technique (k-AESA) for fast and accurate string classification , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[2]  V. Adithya,et al.  Artificial neural network based method for Indian sign language recognition , 2013, 2013 IEEE CONFERENCE ON INFORMATION AND COMMUNICATION TECHNOLOGIES.

[3]  Gang Hua,et al.  Dynamic hand gesture recognition: An exemplar-based approach from motion divergence fields , 2012, Image Vis. Comput..

[4]  Khaled Assaleh,et al.  Video-based signer-independent Arabic sign language recognition using hidden Markov models , 2009, Appl. Soft Comput..

[5]  Teuvo Kohonen,et al.  Median strings , 1985, Pattern Recognit. Lett..

[6]  Ian D. Reid,et al.  3D hand tracking for human computer interaction , 2012, Image Vis. Comput..

[7]  Symone G. Soares Alcalá,et al.  Static gestures recognition for Brazilian Sign Language with kinect sensor , 2016, 2016 IEEE SENSORS.

[8]  Pusadee Seresangtakul,et al.  Thai finger-spelling sign language recognition using global and local features with SVM , 2017, 2017 9th International Conference on Knowledge and Smart Technology (KST).

[9]  James M. Keller,et al.  Fuzzy Models and Algorithms for Pattern Recognition and Image Processing , 1999 .

[10]  George Awad,et al.  Modelling and segmenting subunits for sign language recognition based on hand motion analysis , 2009, Pattern Recognit. Lett..

[11]  Yung-Hui Lee,et al.  Taiwan sign language (TSL) recognition based on 3D data and neural networks , 2009, Expert Syst. Appl..

[12]  King-Sun Fu,et al.  A Clustering Procedure for Syntactic Patterns , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[13]  Geoffrey E. Hinton,et al.  Glove-Talk: a neural network interface between a data-glove and a speech synthesizer , 1993, IEEE Trans. Neural Networks.

[14]  Hermann Ney,et al.  Combination of Tangent Distance and an Image Distortion Model for Appearance-Based Sign Language Recognition , 2005, DAGM-Symposium.

[15]  Samir I. Shaheen,et al.  Sign language recognition using a combination of new vision based features , 2011, Pattern Recognit. Lett..

[16]  A. Abdelbaky Ahmed,et al.  Appearance-based Arabic Sign Language recognition using Hidden Markov Models , 2014, 2014 International Conference on Engineering and Technology (ICET).

[17]  Pinit Kumhom,et al.  Hand posture classification using wavelet moment invariant , 2004, 2004 IEEE Symposium on Virtual Environments, Human-Computer Interfaces and Measurement Systems, 2004. (VCIMS)..

[18]  Zhongfu Ye,et al.  Sign Language Recognition System Based on Weighted Hidden Markov Model , 2015, 2015 8th International Symposium on Computational Intelligence and Design (ISCID).

[19]  Dimitris N. Metaxas,et al.  Adapting hidden Markov models for ASL recognition by using three-dimensional computer vision methods , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[20]  Kuo-Lung Wu,et al.  Unsupervised possibilistic clustering , 2006, Pattern Recognit..

[21]  Ruiduo Yang,et al.  Coupled grouping and matching for sign and gesture recognition , 2009, Comput. Vis. Image Underst..

[22]  Ching-Hua Chuan,et al.  American Sign Language Recognition Using Leap Motion Sensor , 2014, 2014 13th International Conference on Machine Learning and Applications.

[23]  Stan Sclaroff,et al.  Simultaneous Localization and Recognition of Dynamic Hand Gestures , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[24]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[25]  Chokri Ben Amar,et al.  Arabic sign language recognition system based on wavelet networks , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[26]  M. B. Waldron,et al.  Isolated ASL sign recognition system for deaf persons , 1995 .

[27]  Basilio Sierra,et al.  Layered architecture for real time sign recognition: Hand gesture and movement , 2010, Eng. Appl. Artif. Intell..

[28]  Chung-Lin Huang,et al.  Sign language recognition using model-based tracking and a 3D Hopfield neural network , 1998, Machine Vision and Applications.

[29]  Shayhan Ameen Chowdhury,et al.  Hand sign language recognition for Bangla alphabet using Support Vector Machine , 2016, 2016 International Conference on Innovations in Science, Engineering and Technology (ICISET).

[30]  Daniel Kelly,et al.  A person independent system for recognition of hand postures used in sign language , 2010, Pattern Recognit. Lett..

[31]  Ming Ouhyoung,et al.  A real-time continuous gesture recognition system for sign language , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[32]  Haibo Wang,et al.  Chinese sign language recognition with 3D hand motion trajectories and depth images , 2014, Proceeding of the 11th World Congress on Intelligent Control and Automation.

[33]  Hermann Ney,et al.  Handling multimodality and scarce resources in sign language machine translation , 2016 .

[34]  Francisco Casacuberta,et al.  Topology of Strings: Median String is NP-Complete , 1999, Theor. Comput. Sci..

[35]  Wen Gao,et al.  A Chinese sign language recognition system based on SOFM/SRN/HMM , 2004, Pattern Recognit..

[36]  Mu-Chun Su,et al.  A static hand gesture recognition system using a composite neural network , 1996, Proceedings of IEEE 5th International Fuzzy Systems.

[37]  Peter Vamplew.  Recognition of sign language gestures using neural networks , 1996 .

[38]  Ho-Sub Yoon,et al.  Hand gesture recognition using hidden Markov models , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[39]  Chung-Lin Huang,et al.  Hand gesture recognition using a real-time tracking method and hidden Markov models , 2003, Image Vis. Comput..

[40]  Ming C. Leu,et al.  American Sign Language word recognition with a sensory glove using artificial neural networks , 2011, Eng. Appl. Artif. Intell..

[41]  Oscar Koller,et al.  SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[42]  Hermann Ney,et al.  Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition , 2016, BMVC.

[43]  Sansanee Auephanwiriyakul,et al.  Thai Sign Language Translation Using Fuzzy C-Means and Scale Invariant Feature Transform , 2008, ICCSA.

[44]  Agnès Just,et al.  A comparative study of two state-of-the-art sequence processing techniques for hand gesture recognition , 2009, Comput. Vis. Image Underst..

[45]  T. Kobayashi,et al.  Partly-hidden Markov model and its application to gesture recognition , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[46]  Wu jiangqin,et al.  A simple sign language recognition system based on data glove , 1998, ICSP '98. 1998 Fourth International Conference on Signal Processing (Cat. No.98TH8344).

[47]  Md. Mohiminul Islam,et al.  Real time Hand Gesture Recognition using different algorithms based on American Sign Language , 2017, 2017 IEEE International Conference on Imaging, Vision & Pattern Recognition (icIVPR).

[48]  Sansanee Auephanwiriyakul,et al.  Thai sign language translation using Scale Invariant Feature Transform and Hidden Markov Models , 2013, Pattern Recognit. Lett..

[49]  Sansanee Auephanwiriyakul,et al.  A string grammar fuzzy-possibilistic C-medians , 2017, Appl. Soft Comput..

[50]  Kirsti Grobel,et al.  Isolated sign language recognition using hidden Markov models , 1996, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[51]  M. Deriche,et al.  Arabic sign language recognition by decisions fusion using Dempster-Shafer theory of evidence , 2013, 2013 Computing, Communications and IT Applications Conference (ComComAp).

[52]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  Alan W. C. Tan,et al.  Block-based histogram of optical flow for isolated sign language recognition , 2016, J. Vis. Commun. Image Represent..

[54]  Eun-Jung Holden,et al.  Adaptive Fuzzy Expert System for Sign Recognition , 1999 .

[55]  Francisco Casacuberta,et al.  Use of median string for classification , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[56]  Rini Akmeliawati,et al.  Gaussian Process Dynamical Models for hand gesture interpretation in Sign Language , 2011, Pattern Recognit. Lett..

[57]  James M. Keller,et al.  A fuzzy K-nearest neighbor algorithm , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[58]  Soma Mitra,et al.  Vision based Hand Gesture Recognition using Dynamic Time Warping for Indian Sign Language , 2016, 2016 International Conference on Information Science (ICIS).

[59]  King-Sun Fu,et al.  Syntactic Pattern Recognition And Applications , 1968 .