Sign language recognition using sub-units

This paper discusses sign language recognition using linguistic sub-units. It presents three types of sub-units for consideration; those learnt from appearance data as well as those inferred from both 2D or 3D tracking data. These sub-units are then combined using a sign level classifier; here, two options are presented. The first uses Markov Models to encode the temporal changes between sub-units. The second makes use of Sequential Pattern Boosting to apply discriminative feature selection at the same time as encoding temporal information. This approach is more robust to noise and performs well in signer independent tests, improving results from the 54% achieved by the Markov Chains to 76%.

[1]  George Awad,et al.  Modelling and segmenting subunits for sign language recognition based on hand motion analysis , 2009, Pattern Recognit. Lett..

[2]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[3]  Kent Lyons,et al.  GART: The Gesture and Activity Recognition Toolkit , 2007, HCI.

[4]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[5]  Scott K. Liddell,et al.  American Sign Language: The Phonological Base , 2013 .

[6]  Helen Cooper,et al.  University of Surrey , 2019, The Grants Register 2022.

[7]  Alex Pentland,et al.  Real-time American Sign Language recognition from video using hidden Markov models , 1995 .

[8]  Andrew Zisserman,et al.  Learning sign language by watching TV (using weakly aligned subtitles) , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[10]  James M. Rehg,et al.  Learning the basic units in American Sign Language using discriminative segmental feature selection , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[11]  Mairian Corker,et al.  Dictionary of British Sign Language/English , 1993 .

[12]  W. Stokoe,et al.  Sign language structure: an outline of the visual communication systems of the American deaf. 1960. , 1961, Journal of deaf studies and deaf education.

[13]  Thad Starner,et al.  American sign language recognition with the kinect , 2011, ICMI '11.

[14]  Richard Bowden,et al.  Learning Sequential Patterns for Lipreading , 2011, BMVC.

[15]  Andrew Zisserman,et al.  Minimal Training, Large Lexicon, Unconstrained Sign Language Recognition , 2004, BMVC.

[16]  Richard Bowden,et al.  Large Lexicon Detection of Sign Language , 2007, ICCV-HCI.

[17]  M. B. Waldron,et al.  Adaptation of self organizing network for ASL recognition , 1993, Proceedings of the 15th Annual International Conference of the IEEE Engineering in Medicine and Biology Societ.

[18]  Petros Maragos,et al.  Advances in phonetics-based sub-unit modeling for transcription alignment and sign language recognition , 2011, CVPR 2011 WORKSHOPS.

[19]  Dimitris N. Metaxas,et al.  Parallel hidden Markov models for American sign language recognition , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[20]  M. B. Waldron,et al.  Isolated ASL sign recognition system for deaf persons , 1995 .

[21]  M. B. Waldron,et al.  Parsing method for signed telecommunication , 1989, Images of the Twenty-First Century. Proceedings of the Annual International Engineering in Medicine and Biology Society,.

[22]  Yali Amit,et al.  Shape Quantization and Recognition with Randomized Trees , 1997, Neural Computation.

[23]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[24]  Ming-Kuei Hu,et al.  Visual pattern recognition by moment invariants , 1962, IRE Trans. Inf. Theory.

[25]  Surendra Ranganath,et al.  Automatic hand trajectory segmentation and phoneme transcription for sign language , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[26]  Petros Maragos,et al.  Hand Tracking and Affine Shape-Appearance Handshape Sub-units in Continuous Sign Language Recognition , 2010, ECCV Workshops.

[27]  Dimitris N. Metaxas,et al.  Adapting hidden Markov models for ASL recognition by using three-dimensional computer vision methods , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[28]  M. B. Waldron,et al.  Increasing manual sign recognition vocabulary through relabelling , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).