Chinese Sign Language Recognition Based on DTW-Distance-Mapping Features

Sign language is an important communication tool between the deaf and the external world. As the number of the Chinese deaf accounts for 15% of the world, it is highly urgent to develop a Chinese sign language recognition (CSLR) system. Recently, a novel phonology- and radical-coded CSL, taking advantages of a limited and constant number of coded gestures, has been preliminarily verified to be feasible for practical CSLR systems. The keynote of this version of CSL is that the same coded gesture performed in different orientations has different meanings. In this paper, we mainly propose a novel two-stage feature representation method to effectively characterize the CSL gestures. First, an orientation-sensitive feature is extracted regarding the distances between the palm center and the key points of the hand contour. Second, the extracted features are transformed by a dynamic time warping- (DTW-) based feature mapping approach for better representation. Experimental results demonstrate the effectiveness of the proposed feature extraction and mapping approaches. The averaged classification accuracy of all the 39 types of CSL gestures acquired from 11 subjects exceeds 93% for all the adopted classifiers, achieving significant improvement compared to the scheme without DTW-distance-mapping.

[1]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[2]  Pavlo Molchanov,et al.  Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Bo Liu,et al.  Static hand gesture recognition based on finger root-center-angle and length weighted Mahalanobis distance , 2016, Photonics Europe.

[4]  Zaid Omar,et al.  A review of hand gesture and sign language recognition techniques , 2017, International Journal of Machine Learning and Cybernetics.

[5]  Nicolas D. Georganas,et al.  Real-Time Hand Gesture Detection and Recognition Using Bag-of-Features and Support Vector Machine Techniques , 2011, IEEE Transactions on Instrumentation and Measurement.

[6]  Zhongfu Ye,et al.  Continuous sign language recognition using level building based on fast hidden Markov model , 2016, Pattern Recognit. Lett..

[7]  Junsong Yuan,et al.  Robust hand gesture recognition based on finger-earth mover's distance with a commodity depth camera , 2011, ACM Multimedia.

[8]  Pavel Pudil,et al.  Introduction to Statistical Pattern Recognition , 2006 .

[9]  Nikolaos G. Bourbakis,et al.  A survey of skin-color modeling and detection methods , 2007, Pattern Recognit..

[10]  Zhongfu Ye,et al.  A Novel Chinese Sign Language Recognition Method Based on Keyframe-Centered Clips , 2018, IEEE Signal Processing Letters.

[11]  Marcos Faúndez-Zanuy,et al.  On-line signature recognition based on VQ-DTW , 2007, Pattern Recognit..

[12]  Philip Chan,et al.  Toward accurate dynamic time warping in linear time and space , 2007, Intell. Data Anal..

[13]  Shoushui Wei,et al.  [Biometric identification method for ECG based on the piecewise linear representation (PLR) and dynamic time warping (DTW)]. , 2013, Sheng wu yi xue gong cheng xue za zhi = Journal of biomedical engineering = Shengwu yixue gongchengxue zazhi.

[14]  Hyo-Rim Choi,et al.  Modified Dynamic Time Warping Based on Direction Similarity for Fast Gesture Recognition , 2018 .

[15]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[16]  Wen Gao,et al.  Synthetic data generation technique in Signer-independent sign language recognition , 2009, Pattern Recognit. Lett..

[17]  F. Itakura,et al.  Minimum prediction residual principle applied to speech recognition , 1975 .

[18]  Kongqiao Wang,et al.  A Sign-Component-Based Framework for Chinese Sign Language Recognition Using Accelerometer and sEMG Data , 2012, IEEE Transactions on Biomedical Engineering.

[19]  Wen Gao,et al.  Large vocabulary sign language recognition based on fuzzy decision trees , 2004, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[20]  Xilin Chen,et al.  Sparse Observation (SO) Alignment for Sign Language Recognition , 2016, Neurocomputing.

[21]  M. Mohandes,et al.  Image-Based and Sensor-Based Approaches to Arabic Sign Language Recognition , 2014, IEEE Transactions on Human-Machine Systems.

[22]  Wu-Chih Hu,et al.  Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination , 2011, Expert Syst. Appl..

[23]  Jing-Yu Yang,et al.  Face recognition based on the uncorrelated discriminant transformation , 2001, Pattern Recognit..

[24]  Prabin Kumar Bora,et al.  A robust static hand gesture recognition system using geometry based normalizations and Krawtchouk moments , 2013, Pattern Recognit..

[25]  Chun-Jen Chen,et al.  A linear-time component-labeling algorithm using contour tracing technique , 2004, Comput. Vis. Image Underst..

[26]  Olufemi A. Omitaomu,et al.  Weighted dynamic time warping for time series classification , 2011, Pattern Recognit..

[27]  Donald J. Berndt,et al.  Using Dynamic Time Warping to Find Patterns in Time Series , 1994, KDD Workshop.

[28]  Junsong Yuan,et al.  Robust Part-Based Hand Gesture Recognition Using Kinect Sensor , 2013, IEEE Transactions on Multimedia.

[29]  Yun Li,et al.  Surface electromyography and acceleration based Sign Language Recognition using Hidden Conditional Random Fields , 2012, 2012 IEEE-EMBS Conference on Biomedical Engineering and Sciences.

[30]  Chin-Shyurng Fahn,et al.  A Human-Machine Interaction Technique: Hand Gesture Recognition Based on Hidden Markov Models with Trajectory of Hand Motion , 2011 .

[31]  Peng Guo,et al.  Multimodal Fusion Based on LSTM and a Couple Conditional Hidden Markov Model for Chinese Sign Language Recognition , 2019, IEEE Access.

[32]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Aiping Liu,et al.  A Novel Phonology- and Radical-Coded Chinese Sign Language Recognition Framework Using Accelerometer and Surface Electromyography Sensors , 2015, Sensors.

[34]  Eamonn J. Keogh,et al.  Derivative Dynamic Time Warping , 2001, SDM.

[35]  A. Ganapathiraju,et al.  LINEAR DISCRIMINANT ANALYSIS - A BRIEF TUTORIAL , 1995 .

[36]  Somaya Adwan,et al.  A new approach for image stitching technique using Dynamic Time Warping (DTW) algorithm towards scoliosis X-ray diagnosis , 2016 .

[37]  Hong Cheng,et al.  An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition , 2016, Pattern Recognit..

[38]  Fulin Wei,et al.  Position-independent gesture recognition using sEMG signals via canonical correlation analysis , 2018, Comput. Biol. Medicine.

[39]  Marco Cuturi,et al.  Soft-DTW: a Differentiable Loss Function for Time-Series , 2017, ICML.

[40]  Yu Hu,et al.  Body-Earth Mover’s Distance: A Matching-Based Approach for Sleep Posture Recognition , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[41]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Houqiang Li,et al.  A new system for Chinese sign language recognition , 2015, 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP).

[43]  Rami N. Khushaba,et al.  Correlation Analysis of Electromyogram Signals for Multiuser Myoelectric Interfaces , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[44]  Chris Harris,et al.  Geometry from visual motion , 1993 .

[45]  Myoungho Lee,et al.  Adaptive threshold method for the peak detection of photoplethysmographic waveform , 2009, Comput. Biol. Medicine.

[46]  Qing Zhu,et al.  Video-based Chinese sign language recognition using convolutional neural network , 2017, 2017 IEEE 9th International Conference on Communication Software and Networks (ICCSN).

[47]  Robyn A. Owens,et al.  Australian sign language recognition , 2005, Machine Vision and Applications.

[48]  Yaacob Sazali,et al.  Classification of human emotion from EEG using discrete wavelet transform , 2010 .

[49]  Ruchika Lamba,et al.  Automatic Face Detection Using Color Based Segmentation , 2012 .

[50]  Derek D. Lichti,et al.  Towards Real-Time and Rotation-Invariant American Sign Language Alphabet Recognition Using a Range Camera , 2012, Sensors.