Reading the signs: A video based sign dictionary

This article presents a dictionary for Sign Language using visual sign recognition based on linguistic sub-components. We demonstrate a system where the user makes a query, receiving in response a ranked selection of similar results. The approach uses concepts from linguistics to provide sign sub-unit features and classifiers based on motion, sign-location and handshape. These sub-units are combined using Markov Models for sign level recognition. Results are shown for a video dataset of 984 isolated signs performed by a native signer. Recognition rates reach 71.4% for the first candidate and 85.9% for retrieval within the top 10 ranked signs.

[1]  Petros Maragos,et al.  Hand Tracking and Affine Shape-Appearance Handshape Sub-units in Continuous Sign Language Recognition , 2010, ECCV Workshops.

[2]  Andrew Zisserman,et al.  Minimal Training, Large Lexicon, Unconstrained Sign Language Recognition , 2004, BMVC.

[3]  Vassilis Athitsos,et al.  A System for Large Vocabulary Sign Search , 2010, ECCV Workshops.

[4]  Ceil Lucas,et al.  Linguistics of American Sign Language: An Introduction , 1995 .

[5]  Scott K. Liddell,et al.  American Sign Language: The Phonological Base , 2013 .

[6]  Andrew Zisserman,et al.  Learning sign language by watching TV (using weakly aligned subtitles) , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Mairian Corker,et al.  Dictionary of British Sign Language/English , 1993 .

[8]  Petros Maragos,et al.  Advances in phonetics-based sub-unit modeling for transcription alignment and sign language recognition , 2011, CVPR 2011 WORKSHOPS.

[9]  Yali Amit,et al.  Shape Quantization and Recognition with Randomized Trees , 1997, Neural Computation.

[10]  James M. Rehg,et al.  Learning the basic units in American Sign Language using discriminative segmental feature selection , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[11]  W. Stokoe,et al.  Sign language structure: an outline of the visual communication systems of the American deaf. 1960. , 1961, Journal of deaf studies and deaf education.

[12]  Vassilis Athitsos,et al.  A Similarity Measure for Vision-Based Sign Recognition , 2009, HCI.

[13]  Bencie Woll,et al.  The Linguistics of British Sign Language: An Introduction , 1999 .

[14]  George Awad,et al.  Modelling and segmenting subunits for sign language recognition based on hand motion analysis , 2009, Pattern Recognit. Lett..

[15]  H. Ney,et al.  Towards Automatic Sign Language Annotation for the ELAN Tool , 2008 .

[16]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.