Video-based hand manipulation capture through composite motion control

This paper describes a new method for acquiring physically realistic hand manipulation data from multiple video streams. The key idea of our approach is to introduce a composite motion control to simultaneously model hand articulation, object movement, and subtle interaction between the hand and object. We formulate video-based hand manipulation capture in an optimization framework by maximizing the consistency between the simulated motion and the observed image data. We search an optimal motion control that drives the simulation to best match the observed image data. We demonstrate the effectiveness of our approach by capturing a wide range of high-fidelity dexterous manipulation data. We show the power of our recovered motion controllers by adapting the captured motion data to new objects with different properties. The system achieves superior performance against alternative methods such as marker-based motion capture and kinematic hand motion tracking.

[1]  Kenneth Kreutz-Delgado,et al.  Super-Gaussian Mixture Source Model for ICA , 2006, ICA.

[2]  Jessica K. Hodgins,et al.  Video-based 3D motion capture through biped control , 2012, ACM Trans. Graph..

[3]  M. V. D. Panne,et al.  Sampling-based contact-rich motion control , 2010, ACM Trans. Graph..

[4]  Eugene Fiume,et al.  Helping hand: an anatomically accurate inverse dynamics solution for unconstrained hand motion , 2005, SCA '05.

[5]  David J. Fleet,et al.  Model-Based 3D Hand Pose Estimation from Monocular Video , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  C. Karen Liu,et al.  Eurographics/ Acm Siggraph Symposium on Computer Animation (2008) Synthesis of Interactive Hand Manipulation , 2022 .

[7]  Jovan Popovic,et al.  Real-time hand-tracking with a color glove , 2009, SIGGRAPH '09.

[8]  Ying Wu,et al.  Capturing natural hand articulation , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[9]  Dinesh K. Pai,et al.  Musculotendon simulation for hand animation , 2008, ACM Trans. Graph..

[10]  Takeo Kanade,et al.  Model-based tracking of self-occluding articulated objects , 1995, Proceedings of IEEE International Conference on Computer Vision.

[11]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[13]  Jinxiang Chai,et al.  Combining marker-based mocap and RGB-D camera for acquiring high-fidelity hand motion data , 2012, SCA '12.

[14]  Xiaolin K. Wei,et al.  VideoMocap: modeling physically realistic human motion from monocular video sequences , 2010, ACM Trans. Graph..

[15]  DaiQionghai,et al.  Video-based hand manipulation capture through composite motion control , 2013 .

[16]  Jinxiang Chai,et al.  VideoMocap: modeling physically realistic human motion from monocular video sequences , 2010, SIGGRAPH 2010.

[17]  Jovan Popović,et al.  Real-time hand-tracking with a color glove , 2009, SIGGRAPH 2009.

[18]  Thomas S. Huang,et al.  Tracking articulated hand motion with eigen dynamics analysis , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[19]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[20]  Danica Kragic,et al.  Hands in action: real-time 3D reconstruction of hands in interaction with objects , 2010, 2010 IEEE International Conference on Robotics and Automation.

[21]  Sung-Hee Lee,et al.  Ground reaction force control at each foot: A momentum-based humanoid balance controller for non-level and non-stationary ground , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[22]  Dinesh K. Pai,et al.  Interaction capture and synthesis , 2005, SIGGRAPH 2005.

[23]  Zhengyou Zhang,et al.  Flexible camera calibration by viewing a plane from unknown orientations , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[24]  Bodo Rosenhahn,et al.  Human Motion: Understanding, Modelling, Capture, and Animation (Computational Imaging and Vision) , 2007 .

[25]  Odest Chadwicke Jenkins,et al.  Physical simulation for probabilistic motion tracking , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  C. Karen Liu,et al.  Dextrous manipulation from a grasping pose , 2009, ACM Trans. Graph..

[27]  Victor B. Zordan,et al.  Physically based grasping control from example , 2005, SCA '05.

[28]  Hans-Peter Seidel,et al.  An Introduction to Interacting Simulated Annealing , 2006, Human Motion.

[29]  Ludovic Hoyet,et al.  Sleight of hand: perception of finger motion from reduced marker sets , 2012, I3D '12.

[30]  Stan Sclaroff,et al.  Estimating 3D hand pose from a cluttered image , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[31]  Bodo Rosenhahn,et al.  Human Motion - Understanding, Modeling, Capture and Animation, Second Workshop, Human Motion 2007, Rio de Janeiro, Brazil, October 20, 2007, Proceedings , 2007, Workshop on Human Motion.

[32]  Bodo Rosenhahn,et al.  Proceedings of the 2nd conference on Human motion: understanding, modeling, capture and animation , 2007 .

[33]  Antonis A. Argyros,et al.  Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints , 2011, 2011 International Conference on Computer Vision.

[34]  David J. Fleet,et al.  The Kneed Walker for human pose tracking , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Luc Van Gool,et al.  Motion Capture of Hands in Action Using Discriminative Salient Points , 2012, ECCV.

[36]  Richard W. Cottle,et al.  Linear Complementarity Problem , 2009, Encyclopedia of Optimization.

[37]  C. Karen Liu,et al.  Synthesis of detailed hand manipulations using contact sampling , 2012, ACM Trans. Graph..

[38]  Dinesh K. Pai,et al.  Interaction capture and synthesis , 2005, ACM Trans. Graph..