Real-time sign language letter and word recognition from depth data

In this work, we present a system for recognizing letters and finger-spelled words of the American sign language (ASL) in real-time. To this end, the system segments the hand and estimates the hand orientation from captured depth data. The letter classification is based on average neighborhood margin maximization and relies on the segmented depth data of the hands. For word recognition, the letter confidences are aggregated. Furthermore, the word recognition is used to improve the letter recognition by updating the training examples of the letter classifiers on-line.

[1]  Tomaso A. Poggio,et al.  A general framework for object detection , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[2]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Andrew W. Fitzgibbon,et al.  Real-time gesture recognition using deterministic boosting , 2002, BMVC.

[4]  John K. Tsotsos,et al.  Hand Gesture Recognition within a Linguistics-Based Framework , 2004, ECCV.

[5]  Xia Liu,et al.  Hand gesture recognition using depth data , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[6]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Eduardo Zalama Casanova,et al.  Hand Gesture Recognition for Deaf People Interfacing , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[8]  Ulrich Neumann,et al.  Real-time Hand Pose Recognition Using Low-Resolution Depth Images , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[9]  Stefan Müller,et al.  Hand Gesture Recognition with a Novel IR Time-of-Flight Range Camera-A Pilot Study , 2007, MIRAGE.

[10]  Munib Qutaishat,et al.  American sign language (ASL) recognition based on Hough transform and neural networks , 2007, Expert Syst. Appl..

[11]  Fei Wang,et al.  Feature Extraction by Maximizing the Average Neighborhood Margin , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Hong Yan,et al.  Sign Language Finger Alphabet Recognition from Gabor-PCA Representation of Hand Gestures , 2007, 2007 International Conference on Machine Learning and Cybernetics.

[13]  Joachim Hornegger,et al.  Robust real-time 3D time-of-flight based gesture navigation , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[14]  Joachim Hornegger,et al.  3-D gesture-based scene navigation in medical imaging applications using Time-of-Flight cameras , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[15]  Joachim Hornegger,et al.  Gesture recognition with a Time-Of-Flight camera , 2008, Int. J. Intell. Syst. Technol. Appl..

[16]  Joanna Marnik.  The Polish Finger Alphabet Hand Postures Recognition Using Elastic Graph Matching , 2008, Computer Recognition Systems 2.

[17]  Stephan Liwicki,et al.  Automatic recognition of fingerspelled words in British Sign Language , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[18]  James M. Rehg,et al.  Learning the basic units in American Sign Language using discriminative segmental feature selection , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[19]  Luc Van Gool,et al.  Real-Time Body Pose Recognition Using 2D or 3D Haarlets , 2009, International Journal of Computer Vision.

[20]  Stan Sclaroff,et al.  Sign Language Spotting with a Threshold Model Based on Conditional Random Fields , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Zhi Li,et al.  Real time Hand Gesture Recognition using a Range Camera , 2009, ICRA 2009.

[22]  Hervé Lahamy,et al.  REAL-TIME HAND GESTURE RECOGNITION USING RANGE CAMERAS , 2010 .

[23]  Hermann Ney,et al.  The SignSpeak Project - Bridging the Gap Between Signers and Speakers , 2010, LREC.

[24]  Thad Starner,et al.  A novel approach to American Sign Language (ASL) phrase verification using reversed signing , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[25]  Daniel Kelly,et al.  A person independent system for recognition of hand postures used in sign language , 2010, Pattern Recognit. Lett..

[26]  Petros Maragos,et al.  Model-level data-driven sub-units for signs in videos of continuous Sign Language , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[27]  Yasue Mitsukura,et al.  Classification of hand postures based on 3D vision model for human-robot interaction , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[28]  Luc Van Gool,et al.  Combining RGB and ToF cameras for real-time 3D hand gesture interaction , 2011, WACV.