An ego-camera based finger-spelling recognition system

This paper describes a portable system for finger-spelling recognition, employing a PC and a video taken by an ego-camera mounted on the body of a person who does finger-spelling. The system is intended to be a useful tool for an orally impaired person to communicate to anyone at any place by carrying it with him/her. The images of the finger-spelling hand of a user who is carrying the system is captured by an ego-camera. The hand is extracted from arbitrary backgrounds by use of a Gaussian mixture model and skin color evaluation: The trimmed and normalized image of the extracted hand is recognized employing the feature space defined by applying the principal component analysis to the learning data containing 45 finger-spelled Japanese Hiragana letters each with 50 samples and the nearest neighbor method. The on-line performance of the proposed system is experimentally shown.

[1]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Gregory Shakhnarovich,et al.  Fingerspelling Recognition with Semi-Markov Conditional Random Fields , 2013, 2013 IEEE International Conference on Computer Vision.

[3]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[4]  James W. Davis,et al.  The Recognition of Human Movement Using Temporal Templates , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Stephan Liwicki,et al.  Automatic recognition of fingerspelled words in British Sign Language , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[6]  Edwin Escobedo,et al.  Finger Spelling Recognition from Depth data using Direction Cosines and Histogram of Cumulative Magnitudes , 2015 .

[7]  Thad Starner,et al.  American sign language recognition in game development for deaf children , 2006, Assets '06.

[8]  Paul Lukowicz,et al.  Using multiple sensors for mobile sign language recognition , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..

[9]  Joo Kooi Tan,et al.  Detecting moving objects from a video taken by a moving camera using sequential inference of background images , 2014, Artificial Life and Robotics.

[10]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  Ahmed M. Elgammal,et al.  Online Moving Camera Background Subtraction , 2012, ECCV.

[12]  Nicolas Pugeault,et al.  Spelling it out: Real-time ASL fingerspelling recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).