Jointly network: a network based on CNN and RBM for gesture recognition

Hand belongs to non-rigid objects and is rich in variety, making gesture recognition more difficult. The essence of dynamic gesture recognition is the classification and recognition of single-frame still images. Therefore, this paper mainly focuses on static gesture recognition. At present, there are some problems in gesture recognition, such as accuracy, real-time or poor robustness. To solve the above problems, in this paper, the Kinect sensor is used to obtain the color and depth gesture samples, and the gesture samples are processed. On this basis, a jointly network of CNN and RBM is proposed for gesture recognition. It mainly uses superposed network of multiple RBMs to carry out unsupervised feature extraction and combined with supervised feature extraction of CNN. Finally, these two features are combined to classify them. The simulation results show that the proposed jointly network has a better performance in identifying simple background gesture samples and the recognition capability of gesture samples in complex background needs to be improved.

[1]  Gongfa Li,et al.  A simulation-based study of dispatching rules in a dynamic job shop scheduling problem with batch release and extended technical precedence constraints , 2017, Eur. J. Oper. Res..

[2]  Anupam Agrawal,et al.  Vision based hand gesture recognition for human computer interaction: a survey , 2012, Artificial Intelligence Review.

[3]  Mohan M. Trivedi,et al.  Hand Gesture Recognition in Real Time for Automotive Interfaces: A Multimodal Vision-Based Approach and Evaluations , 2014, IEEE Transactions on Intelligent Transportation Systems.

[4]  Luca Benini,et al.  Gesture Recognition Using Wearable Vision Sensors to Enhance Visitors’ Museum Experiences , 2015, IEEE Sensors Journal.

[5]  Gongfa Li,et al.  Research on steady-state simulation in dynamic job shop scheduling problem , 2015 .

[6]  Tijmen Tieleman,et al.  Training restricted Boltzmann machines using approximations to the likelihood gradient , 2008, ICML '08.

[7]  Gongfa Li,et al.  Numerical simulation of temperature field and thermal stress field in the new type of ladle with the nanometer adiabatic material , 2015 .

[8]  Honghai Liu,et al.  Gesture recognition based on sparse representation , 2016, Int. J. Wirel. Mob. Comput..

[9]  Gongfa Li,et al.  Intelligent Control of Air Compressor Production Process , 2013 .

[10]  Honghai Liu,et al.  Gesture recognition based on an improved local sparse representation classification algorithm , 2017, Cluster Computing.

[11]  Gongfa Li,et al.  Influence of Working Lining Parameters on Temperature and Stress Field of Ladle , 2013 .

[12]  Filiberto Pla,et al.  Three-Dimensional Integral Imaging for Gesture Recognition Under Occlusions , 2017, IEEE Signal Processing Letters.

[13]  Honghai Liu,et al.  Computation of grasping and manipulation for multi-fingered robotic hands , 2015 .

[14]  Gongfa Li,et al.  Development of articulated robot trajectory planning , 2017 .

[15]  Shuncong Zhong,et al.  Sine-modulated wavelength-independent full-range complex spectral optical coherence tomography with an ultra-broadband light source , 2015 .

[16]  Honghai Liu,et al.  Gesture recognition based on sparse representation , 2016 .

[17]  Honghai Liu,et al.  D-S evidential theory on sEMG signal recognition , 2017, Int. J. Comput. Sci. Math..

[18]  Honghai Liu,et al.  Intelligent computation in grasping control of dexterous robot hand , 2015 .

[19]  Geoffrey E. Hinton,et al.  OPTIMAL PERCEPTUAL INFERENCE , 1983 .

[20]  Rabul Hussain Laskar,et al.  Dynamic hand gesture recognition using vision-based approach for human–computer interaction , 2018, Neural Computing and Applications.

[21]  G. Li,et al.  Thermal Mechanical Stress Analysis of Ladle Lining with Integral Brick Joint , 2018 .

[22]  Hui Yu,et al.  Gesture recognition based on binocular vision , 2018, Cluster Computing.

[23]  Isabelle Guyon,et al.  Principal motion components for one-shot gesture recognition , 2017, Pattern Analysis and Applications.

[24]  Ying Sun,et al.  Gear reducer optimal design based on computer multimedia simulation , 2018, The Journal of Supercomputing.

[25]  Honghai Liu,et al.  Fusion hand gesture segmentation and extraction based on CMOS sensor and 3D sensor , 2017 .

[26]  Ugur Güdükbay,et al.  A hand gesture recognition technique for human-computer interaction , 2015, J. Vis. Commun. Image Represent..

[27]  Farbod Razzazi,et al.  Spatio-temporal 3D surface matching for hand gesture recognition using ICP algorithm , 2015, Signal Image Video Process..

[28]  Honghai Liu,et al.  Gesture Recognition Based on Kinect and sEMG Signal Fusion , 2018, Mob. Networks Appl..

[29]  Honghai Liu,et al.  Simultaneous Calibration: A Joint Optimization Approach for Multiple Kinect and External Cameras , 2017, Sensors.

[30]  Dzulkifli Mohamad,et al.  Fused features mining for depth-based hand gesture recognition to classify blind human communication , 2017, Neural Computing and Applications.

[31]  Juan José Pantrigo,et al.  Convolutional Neural Networks and Long Short-Term Memory for skeleton-based human activity and hand gesture recognition , 2018, Pattern Recognit..

[32]  Gongfa Li,et al.  Numerical simulation of the influence factors for rotary kiln in temperature field and stress field and the structure optimization , 2015 .

[33]  Honghai Liu,et al.  Gesture recognition based on modified adaptive orthogonal matching pursuit algorithm , 2017, Cluster Computing.

[34]  Honghai Liu,et al.  Fusion hand gesture segmentation and extraction based on CMOS sensor and 3D sensor , 2017, Int. J. Wirel. Mob. Comput..

[35]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[36]  Ying Sun,et al.  Towards the sEMG hand: internet of things sensors and haptic feedback application , 2018, Multimedia Tools and Applications.

[37]  Chokri Ben Amar,et al.  Facial expression recognition based on a mlp neural network using constructive training algorithm , 2014, Multimedia Tools and Applications.

[38]  Honghai Liu,et al.  Hand gesture recognition based on convolution neural network , 2017, Cluster Computing.

[39]  M. Kalyan Chakravarthi,et al.  Accelerometer Based Static Gesture Recognition and Mobile Monitoring System Using Neural Networks , 2015 .

[40]  Zhe Li,et al.  Adaptive fuzzy sliding mode control algorithm simulation for 2-DOF articulated robot , 2017, Int. J. Wirel. Mob. Comput..

[41]  Gongfa Li,et al.  Coke Oven Intelligent Integrated Control System , 2013 .

[42]  Adnan Khashman,et al.  Deep learning in vision-based static hand gesture recognition , 2017, Neural Computing and Applications.

[43]  Rabul Hussain Laskar,et al.  Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ASCII characters in order to develop a virtual text-entry interface system , 2017, Neural Computing and Applications.

[44]  Honghai Liu,et al.  Development of articulated robot trajectory planning , 2017, Int. J. Comput. Sci. Math..

[45]  Rajkumar Buyya,et al.  Next generation cloud computing: New trends and research directions , 2017, Future Gener. Comput. Syst..

[46]  Gerhard Tröster,et al.  Supporting One-Time Point Annotations for Gesture Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Jianfeng Gao,et al.  Deep stacking networks for information retrieval , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[48]  Gongfa Li,et al.  Research on the method of step feature extraction for EOD robot based on 2D laser radar , 2015 .

[49]  Honghai Liu,et al.  Intelligent control model and its simulation of flue temperature in coke oven , 2015 .

[50]  Honghai Liu,et al.  An Interactive Image Segmentation Method in Hand Gesture Recognition , 2017, Sensors.

[51]  Marie Luvett Goh,et al.  Presentation-Aid Armband with IMU, EMG Sensor and Bluetooth for Free-Hand Writing and Hand Gesture Recognition , 2017 .

[52]  Gongfa Li,et al.  D-S evidential theory on sEMG signal recognition , 2017 .

[53]  Ognjan Luzanin,et al.  Hand gesture recognition using low-budget data glove and cluster-trained probabilistic neural network , 2014 .

[54]  Zhihong Li,et al.  Preparation and characterization of mechanical properties of foam glass for artificial floating island carrier , 2015 .

[55]  Honghai Liu,et al.  Optimal grasp planning of multi-fingered robotic hands:a review , 2015 .

[56]  Martin Saerbeck,et al.  Recent methods and databases in vision-based hand gesture recognition: A review , 2015, Comput. Vis. Image Underst..

[57]  Ioannis Katsavounidis,et al.  Low-Complexity Hand Gesture Recognition System for Continuous Streams of Digits and Letters , 2016, IEEE Transactions on Cybernetics.

[58]  Xiangyang Zhu,et al.  A Multichannel Surface EMG System for Hand Motion Recognition , 2015, Int. J. Humanoid Robotics.

[59]  Bruno J. T. Fernandes,et al.  A dynamic gesture recognition and prediction system using the convexity approach , 2017, Comput. Vis. Image Underst..

[60]  Giancarlo Fortino,et al.  Cloud-based Activity-aaService cyber-physical framework for human activity monitoring in mobility , 2017, Future Gener. Comput. Syst..