Hand Motion and Grasping: Capturing, Recognizing and Synthesizing

In this paper we review the various previous methods in vision-based hand motion analysis and hand motion synthesis and grasping. We present our two solutions to the problem. First, we present an articulated hand pose estimation scheme, in which we improve the regression forest based methods by incorporating high-level hand part feature and hand motion constraints. Second, we present a framework for joint 6D palm pose tracking and hand gesture recognition. Based on the algorithms developed above, we have built several real-time systems to enable human-computer interaction with bare hands, such as virtual object manipulation and hand gesture recognition. We also discuss methods to generate hand motion and grasping movement for Virtual Humans.

[1]  Richard Bowden,et al.  Multi-touchless: Real-time fingertip detection and tracking using geodesic maxima , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[2]  Xiaodong Yang,et al.  Histogram of 3D Facets: A characteristic descriptor for hand gesture recognition , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[3]  Daniel Thalmann,et al.  Model-based hand pose estimation via spatial-temporal hand parsing and 3D fingertip localization , 2013, The Visual Computer.

[4]  Ulrich Neumann,et al.  Real-time Hand Pose Recognition Using Low-Resolution Depth Images , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[5]  Sergio Escalera,et al.  Graph cuts optimization for multi-limb human segmentation in depth maps , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Min Li,et al.  Replicated concurrency control for collaborative feature modelling: A fine granular approach , 2008, Comput. Ind..

[7]  Chong Wang,et al.  Superpixel-Based Hand Gesture Recognition With Kinect Depth Camera , 2015, IEEE Transactions on Multimedia.

[8]  Mark R. Cutkosky,et al.  On grasp choice, grasp models, and the design of hands for manufacturing tasks , 1989, IEEE Trans. Robotics Autom..

[9]  D. Thalmann,et al.  Planning collision-free reaching motions for interactive object manipulation and grasping , 2008, SIGGRAPH '08.

[10]  Michael J. Black,et al.  Recognizing temporal trajectories using the condensation algorithm , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[11]  Fumihito Kyota,et al.  Fast Grasp Synthesis for Various Shaped Objects , 2012, Comput. Graph. Forum.

[12]  Tae-Kyun Kim,et al.  Real-Time Articulated Hand Pose Estimation Using Semi-supervised Transductive Regression Forests , 2013, 2013 IEEE International Conference on Computer Vision.

[13]  Chen Qian,et al.  Realtime and Robust Hand Tracking from Depth , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Wilfried Philips,et al.  Real-time hand tracking by invariant hough forest detection , 2012, 2012 19th IEEE International Conference on Image Processing.

[15]  Lale Akarun,et al.  Real time hand pose estimation using depth sensors , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[16]  Makiko Kouchi,et al.  Optimization-Based Grasp Posture Generation Method of Digital Hand for Virtual Ergonomics Assessment , 2008 .

[17]  Daniel Thalmann,et al.  Planning with Smart Objects , 2005, WSCG.

[18]  Cheng Li,et al.  Pixel-Level Hand Detection in Ego-centric Videos , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[20]  Ivan Poupyrev,et al.  Manipulating Objects in Virtual Worlds: Categorization and Empirical Evaluation of Interaction Techniques , 1999, J. Vis. Lang. Comput..

[21]  Chung-Lin Huang,et al.  Hand gesture recognition using a real-time tracking method and hidden Markov models , 2003, Image Vis. Comput..

[22]  Junsong Yuan,et al.  Robust Part-Based Hand Gesture Recognition Using Kinect Sensor , 2013, IEEE Transactions on Multimedia.

[23]  Daniel Thalmann,et al.  Resolving Ambiguous Hand Pose Predictions by Exploiting Part Correlations , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Antonis A. Argyros,et al.  Efficient model-based 3D tracking of hand articulations using Kinect , 2011, BMVC.

[25]  W. Lewis Johnson,et al.  Steve: an animated pedagogical agent for procedural training in virtual environments , 1997, SGAR.

[26]  J. Rossignac,et al.  Plumber: a method for a multi-scale decomposition of 3D shapes into tubular primitives and bodies , 2004, SM '04.

[27]  Yuyu Xu,et al.  An example-based motion synthesis technique for locomotion and object manipulation , 2012, I3D '12.

[28]  Daniel Thalmann,et al.  Joint-dependent local deformations for hand animation and object grasping , 1989 .

[29]  Daniel Thalmann,et al.  Extended grasping behavior for autonomous human agents , 1997, AGENTS '97.

[30]  Tae-Kyun Kim,et al.  Tensor Canonical Correlation Analysis for Action Classification , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Kang-Hyun Jo,et al.  Manipulative hand gesture recognition using task knowledge for human computer interaction , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[32]  Hans-Peter Seidel,et al.  A data-driven approach for real-time full body pose reconstruction from a depth camera , 2011, 2011 International Conference on Computer Vision.

[33]  Heni Ben Amor,et al.  Grasp synthesis from low‐dimensional probabilistic grasp models , 2008, Comput. Animat. Virtual Worlds.

[34]  Marcelo Kallmann,et al.  Motion Parameterization with Inverse Blending , 2010, MIG.

[35]  Daniel P. Huttenlocher,et al.  Pictorial Structures for Object Recognition , 2004, International Journal of Computer Vision.

[36]  Lale Akarun,et al.  Hierarchically constrained 3D hand pose estimation using regression forests from single frame depth data , 2014, Pattern Recognit. Lett..

[37]  Matei T. Ciocarlie,et al.  Dimensionality reduction for hand-independent dexterous robotic grasping , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[38]  Doug A. Bowman,et al.  Formalizing the Design, Evaluation, and Application of Interaction Techniques for Immersive Virtual Environments , 1999, J. Vis. Lang. Comput..

[39]  Karan Singh,et al.  Eurographics/siggraph Symposium on Computer Animation (2003) Handrix: Animating the Human Hand , 2003 .

[40]  Chris Hand,et al.  A Survey of 3D Interaction Techniques , 1997, Comput. Graph. Forum.

[41]  Norman I. Badler,et al.  Real-Time Inverse Kinematics Techniques for Anthropomorphic Limbs , 2000, Graph. Model..

[42]  Aggelos K. Katsaggelos,et al.  Part-based initialization for hand tracking , 2010, 2010 IEEE International Conference on Image Processing.

[43]  Danica Kragic,et al.  Monocular real-time 3D articulated hand pose estimation , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[44]  Rogério Schmidt Feris,et al.  The isometric self-organizing map for 3D hand pose estimation , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[45]  Haiying Guan,et al.  Model-based 3D hand posture estimation from a single 2D image , 2002, Image Vis. Comput..

[46]  Luc Van Gool,et al.  Motion Capture of Hands in Action Using Discriminative Salient Points , 2012, ECCV.

[47]  Tamim Asfour,et al.  Particle Filter-based Fingertip Tracking with Circular Hough Transform Features , 2011, MVA.

[48]  Sylvain Paris,et al.  6D hands: markerless hand-tracking for computer aided design , 2011, UIST.

[49]  N.D. Georganas,et al.  Real-time Vision-based Hand Gesture Recognition Using Haar-like Features , 2007, 2007 IEEE Instrumentation & Measurement Technology Conference IMTC 2007.

[50]  Norman I. Badler,et al.  Virtual humans for animation, ergonomics, and simulation , 1997, Proceedings IEEE Nonrigid and Articulated Motion Workshop.

[51]  Cristian Sminchisescu,et al.  Iterated Second-Order Label Sensitive Pooling for 3D Human Pose Estimation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Nanning Zheng,et al.  Training more discriminative multi-class classifiers for hand detection , 2015, Pattern Recognit..

[53]  Ying Wu,et al.  3D model-based hand tracking using stochastic direct search method , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[54]  Nicolas Pugeault,et al.  Spelling it out: Real-time ASL fingerspelling recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[55]  Daniel Thalmann,et al.  3D fingertip and palm tracking in depth image sequences , 2012, ACM Multimedia.

[56]  Rich Caruana,et al.  An empirical evaluation of supervised learning in high dimensions , 2008, ICML '08.

[57]  Zhigeng Pan,et al.  Biomechanics-based reaching optimization , 2011, The Visual Computer.

[58]  Jinxiang Chai,et al.  Combining marker-based mocap and RGB-D camera for acquiring high-fidelity hand motion data , 2012, SCA '12.

[59]  William T. Freeman,et al.  Orientation Histograms for Hand Gesture Recognition , 1995 .

[60]  Jinxiang Chai,et al.  Accurate realtime full-body motion capture using a single depth camera , 2012, ACM Trans. Graph..

[61]  Li Cheng,et al.  Efficient Hand Pose Estimation from a Single Depth Image , 2013, 2013 IEEE International Conference on Computer Vision.

[62]  Daniel Thalmann,et al.  Parsing the Hand in Depth Images , 2014, IEEE Transactions on Multimedia.

[63]  Yoichi Sato,et al.  Real-Time Fingertip Tracking and Gesture Recognition , 2002, IEEE Computer Graphics and Applications.

[64]  Stan Sclaroff,et al.  Estimating 3D hand pose from a cluttered image , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[65]  Jian Sun,et al.  Cascaded hand pose regression , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).