American sign language recognition and training method with recurrent neural network

Abstract Though American sign language (ASL) has gained recognition from the American society, few ASL applications have been developed with educational purposes. Those designed with real-time sign recognition systems are also lacking. Leap motion controller facilitates the real-time and accurate recognition of ASL signs. It allows an opportunity for designing a learning application with a real-time sign recognition system that seeks to improve the effectiveness of ASL learning. The project proposes an ASL learning application prototype. The application would be a whack-a-mole game with a real-time sign recognition system embedded. Since both static and dynamic signs (J, Z) exist in ASL alphabets, Long-Short Term Memory Recurrent Neural Network with k-Nearest-Neighbour method is adopted as the classification method is based on handling of sequences of input. Characteristics such as sphere radius, angles between fingers and distance between finger positions are extracted as input for the classification model. The model is trained with 2600 samples, 100 samples taken for each alphabet. The experimental results revealed that the recognition rate for 26 ASL alphabets yields an average of 99.44% accuracy rate and 91.82% in 5-fold cross-validation with the use of leap motion controller.

[1]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Antonio Gulli.  Deep learning with Keras : implement neural networks with Keras on Theano and TensorFlow , 2017 .

[3]  Jemina Napier,et al.  Sign Language in Action , 2016 .

[4]  Sebastián Maldonado,et al.  SVR-FFS: A novel forward feature selection approach for high-frequency time series forecasting using support vector regression , 2020, Expert Syst. Appl..

[5]  Ming C. Leu,et al.  Linguistic properties based on American Sign Language isolated word recognition with artificial neural networks using a sensory glove and motion tracker , 2007, Neurocomputing.

[6]  Kang Ryoung Park,et al.  OR-Skip-Net: Outer residual skip network for skin segmentation in non-ideal situations , 2020, Expert Syst. Appl..

[7]  Lahcen Koutti,et al.  MultiD-CNN: A multi-dimensional feature learning approach based on deep convolutional networks for gesture recognition in RGB-D image sequences , 2020, Expert Syst. Appl..

[8]  Shu-Chien Hsu,et al.  Deep neural network based framework for complex correlations in engineering metrics , 2020, Adv. Eng. Informatics.

[9]  Roozbeh Jafari,et al.  A Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors , 2016, IEEE Journal of Biomedical and Health Informatics.

[10]  Khaled Assaleh,et al.  Glove-Based Continuous Arabic Sign Language Recognition in User-Dependent Mode , 2015, IEEE Transactions on Human-Machine Systems.

[11]  Wenjin Tao,et al.  American Sign Language alphabet recognition using Convolutional Neural Networks with multiview augmentation and inference fusion , 2018, Eng. Appl. Artif. Intell..

[12]  Weili Fang,et al.  Deep learning-based extraction of construction procedural constraints from construction regulations , 2020, Adv. Eng. Informatics.

[13]  Chao Chen,et al.  A novel axle temperature forecasting method based on decomposition, reinforcement learning optimization and neural network , 2020, Adv. Eng. Informatics.

[14]  Scott K. Liddell,et al.  American Sign Language: The Phonological Base , 2013 .

[15]  Roozbeh Jafari,et al.  Wearable Computers for Sign Language Recognition , 2017, Handbook of Large-Scale Distributed Computing in Smart Healthcare.

[16]  Sultan Almotairi,et al.  User-Independent American Sign Language Alphabet Recognition Based on Depth Image and PCANet Features , 2019, IEEE Access.

[17]  Sabri Boughorbel,et al.  Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric , 2017, PloS one.

[18]  Sergei Vassilvitskii,et al.  K-means: algorithms, analyses, experiments , 2007 .

[19]  Teak Wei Chong,et al.  American Sign Language Recognition Using Leap Motion Controller with Machine Learning Approach , 2018, Sensors.

[20]  Aditi Chattopadhyay,et al.  Real-time anomaly detection framework using a support vector regression for the safety monitoring of commercial aircraft , 2020, Adv. Eng. Informatics.

[21]  Boon Giin Lee,et al.  Smart Wearable Hand Device for Sign Language Interpretation System With Sensors Fusion , 2018, IEEE Sensors Journal.

[22]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[23]  Roozbeh Jafari,et al.  Real-time American Sign Language Recognition using wrist-worn motion and surface EMG sensors , 2015, 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[24]  Jerome P. Lynch,et al.  Sensor data reconstruction using bidirectional recurrent neural network with application to bridge monitoring , 2019, Adv. Eng. Informatics.

[25]  C. K. Suzuki,et al.  Flexible Optical Fiber Bending Transducer for Application in Glove-Based Sensors , 2014, IEEE Sensors Journal.

[26]  Ognjan Luzanin,et al.  Hand gesture recognition using low-budget data glove and cluster-trained probabilistic neural network , 2014 .

[27]  Joze Guna,et al.  An Analysis of the Precision and Reliability of the Leap Motion Sensor and Its Suitability for Static and Dynamic Tracking , 2014, Sensors.

[28]  Archana Ghotkar,et al.  Hand Gesture Recognition Using Leap Motion Controller , 2016 .

[29]  B. Reiser,et al.  Estimation of the Youden Index and its Associated Cutoff Point , 2005, Biometrical journal. Biometrische Zeitschrift.

[30]  Ming C. Leu,et al.  American Sign Language word recognition with a sensory glove using artificial neural networks , 2011, Eng. Appl. Artif. Intell..

[31]  Kemal Akyol,et al.  Comparing of deep neural networks and extreme learning machines based on growing and pruning approach , 2020, Expert Syst. Appl..

[32]  Pattaraporn Khuwuthyakorn,et al.  The Effectiveness of the Game-Based Learning System for the Improvement of American Sign Language Using Kinect. , 2017 .

[33]  Jennifer S Beal-Alvarez.  Deaf students' receptive and expressive american sign language skills: comparisons and relations. , 2014, Journal of deaf studies and deaf education.

[34]  Ricardo J. G. B. Campello,et al.  A fuzzy extension of the Rand index and other related indexes for clustering and classification assessment , 2007, Pattern Recognit. Lett..

[35]  Sergio Escalera,et al.  Hand sign language recognition using multi-view hand skeleton , 2020, Expert Syst. Appl..

[36]  Matt Huenerfauth,et al.  Accurate and Accessible Motion-Capture Glove Calibration for Sign Language Data Collection , 2010, TACC.

[37]  G. Leigh,et al.  Teaching Sign Language to Hearing Parents of Deaf Children: An Action Research Process , 2007 .

[38]  Ayan Banerjee,et al.  A Comparison of Techniques for Sign Language Alphabet Recognition Using Armband Wearables , 2019, ACM Trans. Interact. Intell. Syst..

[39]  Luigi Cinque,et al.  Exploiting Recurrent Neural Networks and Leap Motion Controller for the Recognition of Sign Language and Semaphoric Hand Gestures , 2018, IEEE Transactions on Multimedia.

[40]  Changsheng Xu,et al.  Discriminative Exemplar Coding for Sign Language Recognition With Kinect , 2013, IEEE Transactions on Cybernetics.