Thai sign language translation system using upright speed-up robust feature and dynamic time warping

Sign language is an important communication tool for the deaf. There are two types of Thai sign language, i.e., Thai finger spelling (a signer spells each character using their fingers) and Thai sign language (a signer uses hand gesture and face expression to represent each word). In this paper, we build a dynamic hand gesture translation system with video caption without prior hand region segmentation. In particular, we utilize the upright speed-up robust feature (U-SURF) and the dynamic time warping (DTW) to find a matched word. This Thai sign language translation system was tested on 42 words. The total number of video sequences used in the experiment is 1470. The best correct classification rate for blind test set of signer-dependent is approximately 98% whereas that of signer-semi-independent is around 76 to 79%. The best blind test result for the signer-independent is around 62 to 68%.

[1]  Zongda Wu,et al.  Similarity Match in Time Series Streams under Dynamic Time Warping Distance , 2008, 2008 International Conference on Computer Science and Software Engineering.

[2]  Wu jiangqin,et al.  A simple sign language recognition system based on data glove , 1998, ICSP '98. 1998 Fourth International Conference on Signal Processing (Cat. No.98TH8344).

[3]  Byeong Seok Ahn.  An Efficient Elimination of Input Data in the OWA Aggregation , 2007 .

[4]  Sansanee Auephanwiriyakul,et al.  Incorporating SIFT with Hard C-Means algorithm , 2010, 2010 The 2nd International Conference on Computer and Automation Engineering (ICCAE).

[5]  Quan Yang,et al.  Chinese sign language recognition based on video sequence appearance modeling , 2010, 2010 5th IEEE Conference on Industrial Electronics and Applications.

[6]  Pinit Kumhom,et al.  Hand posture classification using wavelet moment invariant , 2004, 2004 IEEE Symposium on Virtual Environments, Human-Computer Interfaces and Measurement Systems, 2004. (VCIMS)..

[7]  Wenyi Zeng,et al.  Clustering Time Series with Granular Dynamic Time Warping Method , 2007, 2007 IEEE International Conference on Granular Computing (GRC 2007).

[8]  Meinard Müller,et al.  Information retrieval for music and motion , 2007 .

[9]  Chung-Lin Huang,et al.  Sign language recognition using model-based tracking and a 3D Hopfield neural network , 1998, Machine Vision and Applications.

[10]  Stan Sclaroff,et al.  Simultaneous Localization and Recognition of Dynamic Hand Gestures , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[11]  Khaled Assaleh,et al.  Video-based signer-independent Arabic sign language recognition using hidden Markov models , 2009, Appl. Soft Comput..

[12]  直樹 水野,et al.  各指の曲げ状態に着目した手話の手形状認識(機械力学,計測,自動制御) , 2002 .

[13]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[14]  Yung-Hui Lee,et al.  Taiwan sign language (TSL) recognition based on 3D data and neural networks , 2009, Expert Syst. Appl..

[15]  Chung-Lin Huang,et al.  Hand gesture recognition using a real-time tracking method and hidden Markov models , 2003, Image Vis. Comput..

[16]  Sansanee Auephanwiriyakul,et al.  Thai Sign Language Translation Using Fuzzy C-Means and Scale Invariant Feature Transform , 2008, ICCSA.

[17]  T. Kobayashi,et al.  Partly-hidden Markov model and its application to gesture recognition , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[18]  Ho-Sub Yoon,et al.  Hand gesture recognition using hidden Markov models , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[19]  Wu Jiang.  A Hierarchical DGMM Recognizer for Chinese Sign Language Recognition , 2000 .

[20]  Ahmed K. Elmagarmid,et al.  Scene change detection techniques for video database systems , 1998, Multimedia Systems.

[21]  Geoffrey E. Hinton,et al.  Glove-Talk: a neural network interface between a data-glove and a speech synthesizer , 1993, IEEE Trans. Neural Networks.

[22]  R. Akmeliawati,et al.  Real-Time Malaysian Sign Language Translation using Colour Segmentation and Neural Network , 2007, 2007 IEEE Instrumentation & Measurement Technology Conference IMTC 2007.

[23]  Wen Gao,et al.  A Chinese sign language recognition system based on SOFM/SRN/HMM , 2004, Pattern Recognit..

[24]  Mu-Chun Su,et al.  A static hand gesture recognition system using a composite neural network , 1996, Proceedings of IEEE 5th International Fuzzy Systems.

[25]  Matthew A. Brown,et al.  Invariant Features from Interest Point Groups , 2002, BMVC.