American Sign Language Alphabet Recognition Using a Neuromorphic Sensor and an Artificial Neural Network

This paper reports the design and analysis of an American Sign Language (ASL) alphabet translation system implemented in hardware using a Field-Programmable Gate Array. The system process consists of three stages, the first being the communication with the neuromorphic camera (also called Dynamic Vision Sensor, DVS) sensor using the Universal Serial Bus protocol. The feature extraction of the events generated by the DVS is the second part of the process, consisting of a presentation of the digital image processing algorithms developed in software, which aim to reduce redundant information and prepare the data for the third stage. The last stage of the system process is the classification of the ASL alphabet, achieved with a single artificial neural network implemented in digital hardware for higher speed. The overall result is the development of a classification system using the ASL signs contour, fully implemented in a reconfigurable device. The experimental results consist of a comparative analysis of the recognition rate among the alphabet signs using the neuromorphic camera in order to prove the proper operation of the digital image processing algorithms. In the experiments performed with 720 samples of 24 signs, a recognition accuracy of 79.58% was obtained.

[1]  Toby Sharp,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR.

[2]  Lale Akarun,et al.  Randomized decision forests for static and dynamic hand shape classification , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[3]  Rekha Lathi,et al.  Dynamic Vision Sensor Camera Based Bare Hand Gesture Recognition , 2012 .

[4]  Junwei Wang,et al.  Shape matching and classification using height functions , 2012, Pattern Recognit. Lett..

[5]  Hiroomi Hikawa,et al.  Novel FPGA Implementation of Hand Sign Recognition System With SOM–Hebb Classifier , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[6]  Jaime López-Krahe,et al.  An online reversed French Sign Language dictionary based on a learning approach for signs classification , 2015, Pattern Recognit. Lett..

[7]  Nicolas Pugeault,et al.  Spelling it out: Real-time ASL fingerspelling recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[8]  Debi Prosad Dogra,et al.  Coupled HMM-based multi-sensor data fusion for sign language recognition , 2017, Pattern Recognit. Lett..

[9]  Tobi Delbrück,et al.  Real-Time Gesture Interface Based on Event-Driven Processing From Stereo Silicon Retinas , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Kin Fun Li,et al.  A Web-Based Sign Language Translator Using 3D Video Processing , 2011, 2011 14th International Conference on Network-Based Information Systems.

[11]  Derek D. Lichti,et al.  Towards Real-Time and Rotation-Invariant American Sign Language Alphabet Recognition Using a Range Camera , 2012, Sensors.

[12]  John A. Albertini,et al.  Deafness and Hearing Loss , 2010 .

[13]  Feng Ran,et al.  Gesture recognition based on parallel hardware neural network implemented with stochastic logics , 2016, 2016 International Conference on Audio, Language and Image Processing (ICALIP).

[14]  Kimiaki Shirahama,et al.  Shape-based object matching using interesting points and high-order graphs , 2016, Pattern Recognit. Lett..

[15]  John Yen,et al.  Dynamic vision sensor camera based bare hand gesture recognition , 2011, 2011 IEEE Symposium On Computational Intelligence For Multimedia, Signal And Vision Processing.

[16]  Suchin Adhan,et al.  American Sign Language recognition by using 3D geometric invariant feature and ANN classification , 2014, The 7th 2014 Biomedical Engineering International Conference.

[17]  S. Amirhassan Monadjemi,et al.  Rapid hand posture recognition using Adaptive Histogram Template of Skin and hand edge contour , 2010, 2010 6th Iranian Conference on Machine Vision and Image Processing.

[18]  Byungik Ahn,et al.  Real-time video object recognition using convolutional neural network , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[19]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[20]  Tobi Delbrück,et al.  A Low Power, Fully Event-Based Gesture Recognition System , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  M. Sundarambal,et al.  FPGA implementation of hand gesture recognition system using neural networks , 2017, 2017 11th International Conference on Intelligent Systems and Control (ISCO).