Fusion of Inertial and Depth Sensor Data for Robust Hand Gesture Recognition

This paper presents the first attempt at fusing data from inertial and vision depth sensors within the framework of a hidden Markov model for the application of hand gesture recognition. The data fusion approach introduced in this paper is general purpose in the sense that it can be used for recognition of various body movements. It is shown that the fusion of data from the vision depth and inertial sensors act in a complementary manner leading to a more robust recognition outcome compared with the situations when each sensor is used individually on its own. The obtained recognition rates for the single hand gestures in the Microsoft MSR data set indicate that our fusion approach provides improved recognition in real-time and under realistic conditions.

[1]  Jin-Hyung Kim,et al.  An HMM-Based Threshold Model Approach for Gesture Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Jeroen D. Hol,et al.  Sensor Fusion and Calibration of Inertial Sensors, Vision, Ultra-Wideband and GPS , 2011 .

[3]  Nasser Kehtarnavaz,et al.  Real-time human action recognition based on depth motion maps , 2016, Journal of Real-Time Image Processing.

[4]  Michael Harrington,et al.  FlightTracker: a novel optical/inertial tracker for cockpit enhanced vision , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[5]  Ying Wu,et al.  Discriminative subvolume search for efficient action detection , 2009, CVPR.

[6]  Zhaoying Zhou,et al.  A real-time articulated human motion tracking using tri-axis inertial/magnetic sensors package. , 2004, IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[7]  Thomas B. Schön,et al.  Sensor Fusion for Augmented Reality , 2008 .

[8]  Mónica F. Bugallo,et al.  Target Tracking by Particle Filtering in Binary Sensor Networks , 2008, IEEE Transactions on Signal Processing.

[9]  Nasser Kehtarnavaz,et al.  Real-time robust vision-based hand gesture recognition using stereo images , 2013, Journal of Real-Time Image Processing.

[10]  Vasile Gui,et al.  Vision Based Hand Gesture Recognition: A Review , 2013 .

[11]  Roozbeh Jafari,et al.  Low power programmable architecture for periodic activity monitoring , 2013, 2013 ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS).

[12]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[13]  Seiichi Uchida,et al.  A Survey of Elastic Matching Techniques for Handwritten Character Recognition , 2005, IEICE Trans. Inf. Syst..

[14]  Hui Li,et al.  Kinect Based Dynamic Hand Gesture Recognition Algorithm Research , 2012, 2012 4th International Conference on Intelligent Human-Machine Systems and Cybernetics.

[15]  Paolo Bonato,et al.  Monitoring Motor Fluctuations in Patients With Parkinson's Disease Using Wearable Sensors , 2009, IEEE Transactions on Information Technology in Biomedicine.

[16]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[17]  Zhiqiang Zhang,et al.  Ubiquitous human body motion capture using micro-sensors , 2009, 2009 IEEE International Conference on Pervasive Computing and Communications.

[18]  Jorge Dias,et al.  Vision and Inertial Sensor Cooperation Using Gravity as a Vertical Reference , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Lale Akarun,et al.  Real time hand pose estimation using depth sensors , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[20]  Alexander A. Sawchuk,et al.  A customizable framework of body area sensor network for rehabilitation , 2009, 2009 2nd International Symposium on Applied Sciences in Biomedical and Communication Technologies.

[21]  R. S. Jadon,et al.  A REVIEW OF VISION BASED HAND GESTURES RECOGNITION , 2009 .

[22]  Héctor Pomares,et al.  Kinect=IMU? Learning MIMO Signal Mappings to Automatically Translate Activity Recognition Systems across Sensor Modalities , 2012, 2012 16th International Symposium on Wearable Computers.

[23]  Dan Schonfeld,et al.  Video Tracking Based on Sequential Particle Filtering on Graphs , 2011, IEEE Transactions on Image Processing.

[24]  Junsong Yuan,et al.  Robust hand gesture recognition with kinect sensor , 2011, ACM Multimedia.

[25]  Jian Lu,et al.  Real-Time Activity Recognition in Wireless Body Sensor Networks: From Simple Gestures to Complex Activities , 2010, 2010 IEEE 16th International Conference on Embedded and Real-Time Computing Systems and Applications.