Deep Learning Based Hand Gesture Recognition and UAV Flight Controls

Dynamic hand gesture recognition is desired as an alternative means for human-computer interactions. This paper presents a hand gesture recognition system that is designed for the control of flights of unmanned aerial vehicles (UAV). To train the system to recognize designed gestures, skeleton data collected from a Leap Motion Controller are converted to two different data models. As many as 9124 samples of training dataset, 1938 samples of testing dataset are created to train and test the proposed three deep learning neural networks, which are a 2-layer fully connected neural network, a 5-layer fully connected neural network and an 8-layer convolutional neural network. The static testing results show that the 2-layer fully connected neural network achieves an average accuracy of 98.2% on normalized datasets and 11% on raw datasets. The 5-layer fully connected neural network achieves an average accuracy of 95.2% on normalized datasets and 45% on raw datasets. The 8-layers convolutional neural network achieves an average accuracy of 96.2% on normalized datasets and raw datasets. Testing on a drone-kit simulator and a real drone shows that this system is feasible for drone flight controls.

[1]  Jiang Long,et al.  Instrument-Based Noncontact Doppler Radar Vital Sign Detection System Using Heterodyne Digital Quadrature Demodulation Architecture , 2010, IEEE Transactions on Instrumentation and Measurement.

[2]  Jianhua Tao,et al.  Semi-supervised Ladder Networks for Speech Emotion Recognition , 2019, International Journal of Automation and Computing.

[3]  Nicolas Pugeault,et al.  Sign language recognition using sub-units , 2012, J. Mach. Learn. Res..

[4]  Yi Cao,et al.  Multi-layer Contribution Propagation Analysis for Fault Diagnosis , 2019, Int. J. Autom. Comput..

[5]  Ruiduo Yang,et al.  Gesture Recognition using Hidden Markov Models from Fragmented Observations , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[6]  Zhiwu Lu,et al.  Zero-shot Fine-grained Classification by Deep Feature Learning with Semantics , 2017, Int. J. Autom. Comput..

[7]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[8]  Hermann Ney,et al.  Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition , 2016, BMVC.

[9]  Nobuyuki Otsu,et al.  Gesture recognition using auto-regressive coefficients of higher-order local auto-correlation features , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[10]  Xilin Chen,et al.  Two streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[11]  Pavlo Molchanov,et al.  Hand gesture recognition with 3D convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Trevor Darrell,et al.  Hidden Conditional Random Fields for Gesture Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[13]  Lale Akarun,et al.  Real time gestural interface for generic applications , 2005, 2005 13th European Signal Processing Conference.

[14]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[15]  Aditya Ramamoorthy,et al.  Recognition of dynamic hand gestures , 2003, Pattern Recognit..

[16]  Deepanshu Srivastava,et al.  STEP-based Feature Recognition System for B-spline Surface Features , 2018, Int. J. Autom. Comput..

[17]  Changzhi Li,et al.  A Review on Recent Advances in Doppler Radar Sensors for Noncontact Healthcare Monitoring , 2013, IEEE Transactions on Microwave Theory and Techniques.

[18]  Sander Dieleman,et al.  Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video , 2015, International Journal of Computer Vision.

[19]  Alex Pentland,et al.  Real-time American Sign Language recognition from video using hidden Markov models , 1995 .

[20]  Helge J. Ritter,et al.  Visual recognition of continuous hand postures , 2002, IEEE Trans. Neural Networks.

[21]  Xinying Xu,et al.  Deep Learning Based Single Image Super-resolution: A Survey , 2018, International Journal of Automation and Computing.

[22]  Zhenjie Yao,et al.  Applying Deep Learning to Individual and Community Health Monitoring Data: A Survey , 2018, International Journal of Automation and Computing.

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  Yi Li,et al.  Dynamic hand gesture recognition using hidden Markov models , 2012, 2012 7th International Conference on Computer Science & Education (ICCSE).

[25]  Frank Weichert,et al.  Analysis of the Accuracy and Robustness of the Leap Motion Controller , 2013, Sensors.

[26]  Gerhard Rigoll,et al.  High Performance Real-Time Gesture Recognition Using Hidden Markov Models , 1997, Gesture Workshop.

[27]  O. Bimber.  CONTINUOUS 6 DOF GESTURE RECOGNITION : A FUZZY LOGIC APPROACH , 1999 .

[28]  Oya Aran,et al.  VISION BASED SIGN LANGUAGE RECOGNITION: MODELING AND RECOGNIZING ISOLATED SIGNS WITH MANUAL AND NON-MANUAL COMPONENTS , 2008 .

[29]  W. Marsden.  I and J , 2012 .

[30]  Fabio Nelli,et al.  Deep Learning with TensorFlow , 2018 .

[31]  Bhiksha Raj,et al.  Ultrasonic Doppler Sensing in HCI , 2012, IEEE Pervasive Computing.

[32]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Joze Guna,et al.  An Analysis of the Precision and Reliability of the Leap Motion Sensor and Its Suitability for Static and Dynamic Tracking , 2014, Sensors.

[34]  Christian Wolf,et al.  ModDrop: Adaptive Multi-Modal Gesture Recognition , 2014, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Wu-Chih Hu,et al.  Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination , 2011, Expert Syst. Appl..

[36]  Yu Fu,et al.  Gesture Recognition Based on BP Neural Network Improved by Chaotic Genetic Algorithm , 2018, Int. J. Autom. Comput..

[37]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[38]  Nicolas D. Georganas,et al.  Real-Time Hand Gesture Detection and Recognition Using Bag-of-Features and Support Vector Machine Techniques , 2011, IEEE Transactions on Instrumentation and Measurement.

[39]  Gerald Penn,et al.  Convolutional Neural Networks for Speech Recognition , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[40]  Jiacun Wang,et al.  Deep Learning Based Hand Gesture Recognition and UAV Flight Controls , 2020, Int. J. Autom. Comput..

[41]  Ming C. Leu,et al.  Human-Computer Interaction System with Artificial Neural Network Using Motion Tracker and Data Glove , 2005, PReMI.

[42]  Archana Ghotkar,et al.  Dynamic Hand Gesture Recognition using Hidden Markov Model by Microsoft Kinect Sensor , 2016 .