Sign language recognition based on adaptive HMMS with data augmentation

Vision based sign language recognition (SLR) is a challenging task due to the complexity of signs and limited data collection. To improve the recognition precision, this paper proposes an adaptive GMM-based (Gaussian mixture model) HMMs (Hidden Markov Models) framework. We discover that inherent latent states in HMMs are not only related to the number of key gestures and body poses, but also related to the kinds of their translation relationships. We propose adaptive HMMs and obtain the hidden state number for each sign with affinity propagation clustering. Furthermore, to enrich the training dataset, we propose a data augmentation strategy by adding Gaussian random disturbances. Experiments on a vocabulary of 370 signs demonstrate the effectiveness of our proposed method over the comparison algorithms.

[1]  Chao Xie,et al.  Chinese sign language recognition with adaptive HMM , 2016, 2016 IEEE International Conference on Multimedia and Expo (ICME).

[2]  Vassilis Athitsos,et al.  A System for Large Vocabulary Sign Search , 2010, ECCV Workshops.

[3]  David R. Anderson,et al.  Multimodel Inference , 2004 .

[4]  Houqiang Li,et al.  Sign Language Recognition using 3D convolutional neural networks , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[5]  Houqiang Li,et al.  Sign Language Recognition Based on Trajectory Modeling with HMMs , 2016, MMM.

[6]  Tarik Arici,et al.  Gesture Recognition using Skeleton Data with Weighted Dynamic Time Warping , 2013, VISAPP.

[7]  Kouichi Murakami,et al.  Gesture recognition using recurrent neural networks , 1991, CHI.

[8]  Noriaki Kuwahara,et al.  Development of Recognition System of Japanese Sign Language Using 3D Image Sensor , 2013, HCI.

[9]  M. Maraqa,et al.  Recognition of Arabic Sign Language (ArSL) using recurrent neural networks , 2008, 2008 First International Conference on the Applications of Digital Information and Web Technologies (ICADIWT).

[10]  Lu Yang,et al.  Survey on 3D Hand Gesture Recognition , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[11]  Stan Sclaroff,et al.  A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Andrew Zisserman,et al.  Return of the Devil in the Details: Delving Deep into Convolutional Nets , 2014, BMVC.

[13]  Avinash C. Kak,et al.  Proceedings of IEEE International Conference on Multimodel Interfaces, 2002 , 2022 .

[14]  Jie Huang,et al.  Sign language recognition using real-sense , 2015, 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP).

[15]  Changsheng Xu,et al.  Discriminative Exemplar Coding for Sign Language Recognition With Kinect , 2013, IEEE Transactions on Cybernetics.

[16]  Xilin Chen,et al.  Curve Matching from the View of Manifold for Sign Language Recognition , 2014, ACCV Workshops.

[17]  Nicolas Pugeault,et al.  Sign Language Recognition using Sequential Pattern Trees , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Houqiang Li,et al.  A new system for Chinese sign language recognition , 2015, 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP).

[19]  Philip Chan,et al.  Toward accurate dynamic time warping in linear time and space , 2007, Intell. Data Anal..

[20]  Ying Wu,et al.  Mining actionlet ensemble for action recognition with depth cameras , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  H. Bozdogan,et al.  Akaike's Information Criterion and Recent Developments in Information Complexity. , 2000, Journal of mathematical psychology.

[22]  Z. Liu,et al.  A real time system for dynamic hand gesture recognition with a depth sensor , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[23]  Ruiduo Yang,et al.  Segmentation-robust representations, matching, and modeling for sign language , 2011, CVPR 2011 WORKSHOPS.

[24]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[25]  Andrew Zisserman,et al.  Large-scale Learning of Sign Language by Watching TV (Using Co-occurrences) , 2013, BMVC.

[26]  Xilin Chen,et al.  Fast sign language recognition benefited from low rank approximation , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).