Recognition of Hand Gestures Observed by Depth Cameras

We focus on gesture recognition based on 3D information in the form of a point cloud of the observed scene. A descriptor of the scene is built on the basis of a Viewpoint Feature Histogram (VFH). To increase the distinctiveness of the descriptor the scene is divided into smaller 3D cells and VFH is calculated for each of them. A verification of the method on publicly available Polish and American sign language datasets containing dynamic gestures as well as hand postures acquired by a time-of-flight (ToF) camera or Kinect is presented. Results of cross-validation test are given. Hand postures are recognized using a nearest neighbour classifier with city-block distance. For dynamic gestures two types of classifiers are applied: (i) the nearest neighbour technique with dynamic time warping and (ii) hidden Markov models. The results confirm the usefulness of our approach.

[1]  Mauro Donadeo,et al.  Combining multiple depth-based descriptors for hand gesture recognition , 2014, Pattern Recognit. Lett..

[2]  George Awad,et al.  Modelling and segmenting subunits for sign language recognition based on hand motion analysis , 2009, Pattern Recognit. Lett..

[3]  Junsong Yuan,et al.  Robust hand gesture recognition with kinect sensor , 2011, ACM Multimedia.

[4]  Yi Li,et al.  Hand gesture recognition using Kinect , 2012, 2012 IEEE International Conference on Computer Science and Automation Engineering.

[5]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[6]  Tae-Kyun Kim,et al.  Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Jesús Bescós,et al.  Real-time user independent hand gesture recognition from time-of-flight camera video using static and dynamic models , 2011, Machine Vision and Applications.

[8]  Luc Van Gool,et al.  Real-time sign language letter and word recognition from depth data , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[9]  Rui Min Hu,et al.  Real-Time 3D Hand Tracking from Depth Images , 2013 .

[10]  Tae-Kyun Kim,et al.  Latent-Class Hough Forests for 3D Object Detection and Pose Estimation , 2014, ECCV.

[11]  Gary R. Bradski,et al.  Learning OpenCV 3: Computer Vision in C++ with the OpenCV Library , 2016 .

[12]  Nico Blodow,et al.  Learning informative point classes for the acquisition of object model maps , 2008, 2008 10th International Conference on Control, Automation, Robotics and Vision.

[13]  Richard Bowden,et al.  Natural Action Recognition Using Invariant 3D Motion Encoding , 2014, ECCV.

[14]  Steve Young,et al.  The HTK book , 1995 .

[15]  Riccardo Leonardi,et al.  XKin -: eXtendable hand pose and gesture recognition library for kinect , 2012, ACM Multimedia.

[16]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[17]  Gary R. Bradski,et al.  Fast 3D recognition and pose using the Viewpoint Feature Histogram , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[18]  Nicolas Pugeault,et al.  Spelling it out: Real-time ASL fingerspelling recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[19]  Junsong Yuan,et al.  Depth camera based hand gesture recognition and its applications in Human-Computer-Interaction , 2011, 2011 8th International Conference on Information, Communications & Signal Processing.

[20]  Arif Mahmood,et al.  HOPC: Histogram of Oriented Principal Components of 3D Pointclouds for Action Recognition , 2014, ECCV.

[21]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[22]  Pietro Zanuttigh,et al.  Hand gesture recognition with leap motion and kinect devices , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[23]  Jérôme Martin,et al.  An Appearance-Based Approach to Gesture-Recognition , 1997, ICIAP.

[24]  Lale Akarun,et al.  Real time hand pose estimation using depth sensors , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[25]  Wlodzimierz Kasprzak,et al.  Hand gesture recognition based on free-form contours and probabilistic inference , 2012, Int. J. Appl. Math. Comput. Sci..

[26]  Gary R. Bradski,et al.  Learning OpenCV - computer vision with the OpenCV library: software that sees , 2008 .

[27]  Morteza Zahedi,et al.  Robust Sign Language Recognition System Using ToF Depth Cameras , 2011, ArXiv.

[28]  Luc Van Gool,et al.  Combining RGB and ToF cameras for real-time 3D hand gesture interaction , 2011, WACV.

[29]  Serban Oprisescu,et al.  Automatic static hand gesture recognition using ToF cameras , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[30]  Junsong Yuan,et al.  Robust hand gesture recognition based on finger-earth mover's distance with a commodity depth camera , 2011, ACM Multimedia.

[31]  H. M. Cooper,et al.  Sign language recognition : Generalising to more complex corpora , 2010 .

[32]  Janusz Konrad,et al.  A gesture-driven computer interface using Kinect , 2012, 2012 IEEE Southwest Symposium on Image Analysis and Interpretation.

[33]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[34]  Eamonn J. Keogh,et al.  Three Myths about Dynamic Time Warping Data Mining , 2005, SDM.

[35]  Ralf Dörner,et al.  Recognition of Dynamic Hand Gestures with Time-of-Flight Cameras , 2010 .

[36]  Robin R. Murphy,et al.  Hand gesture recognition with depth images: A review , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.

[37]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[38]  Sergios Theodoridis,et al.  Pattern Recognition , 1998, IEEE Trans. Neural Networks.