Australian sign language recognition

This paper presents an automatic Australian sign language (Auslan) recognition system, which tracks multiple target objects (the face and hands) throughout an image sequence and extracts features for the recognition of sign phrases. Tracking is performed using correspondences of simple geometrical features between the target objects within the current and the previous frames. In signing, the face and a hand of a signer often overlap, thus the system needs to segment these for the purpose of feature extraction. Our system deals with the occlusion of the face and a hand by detecting the contour of the foreground moving object using a combination of motion cues and the snake algorithm. To represent signs, features that are invariant to scaling, 2D rotations and signing speed are used for recognition. The features represent the relative geometrical positioning and shapes of the target objects, as well as their directions of motion. These are used to recognise Auslan phrases using Hidden Markov Models. Experiments were conducted using 163 test sign phrases with varying grammatical formations. Using a known grammar, the system achieved over 97% recognition rate on a sentence level and 99% success rate at a word level.

[1]  Yoshiaki Shirai,et al.  Extraction of Hand Features for Recognition of Sign Language Words , 2002 .

[2]  Suat Akyol,et al.  Finding Relevant Image Content for mobile Sign Language Recognition , 2001 .

[3]  Shan Lu,et al.  Color-based hands tracking system for sign language recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[4]  David Windridge,et al.  A Linguistic Feature Vector for the Visual Interpretation of Sign Language , 2004, ECCV.

[5]  T. Kanade,et al.  Color information for region segmentation , 1980 .

[6]  Takio Kurita,et al.  Scale invariant face detection method using higher-order local autocorrelation features extracted from log-polar image , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[7]  Bryan F. J. Manly,et al.  Multivariate Statistical Methods : A Primer , 1986 .

[8]  Cheng-Chew Lim,et al.  Hand and face segmentation using motion and color cues in digital image sequences , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[9]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[10]  Narendra Ahuja,et al.  Recognizing hand gesture using motion trajectories , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[11]  Michael Isard,et al.  Active Contours , 2000, Springer London.

[12]  Eun-Jung Holden,et al.  An animated Auslan Tuition System , 2003 .

[13]  P.C. Woodland,et al.  The 1994 HTK large vocabulary speech recognition system , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[14]  Eun-Jung Holden,et al.  Segmenting Occluded Objects using a Motion Snake , 2004 .

[15]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[16]  Shan Lu,et al.  Recognition of local features for camera-based sign language recognition system , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[17]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[18]  Mubarak Shah,et al.  A Fast algorithm for active contours and curvature estimation , 1992, CVGIP Image Underst..

[19]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[20]  D. F. Morrison,et al.  Multivariate Statistical Methods , 1968 .

[21]  T. Johnston,et al.  A.U.S.L.A.N dictionary : a dictionary of the sign language of the Australian deaf community , 1989 .