Seeing Skin in Reduced Coordinates

We present a skin tracking and reconstruction method that uses a monocular camera and a depth sensor to recover skin sliding motions on the surface of a deforming object. Such depth cameras are widely available. Our key idea is to use a reduced coordinate framework that implicitly constrains skin to conform to the shape of the underlying object when it slides. The skin configuration in 3D can then be efficiently reconstructed by tracking two dimensional skin features in video. This representation is well suited for tracking subtle skin movements in the upper face and on the hand. The reconstructed skin motions have many uses, including synthesizing and retargeting animations, recognizing facial expressions, and for learning datadriven models of skin movement. In our face tracking examples, we recover subtle but important details of skin movement around the eyes. We validated the algorithm using a hand gesture sequence with known skin motion, recovering skin sliding motion with a low reconstruction error.

[1]  Jitendra Malik,et al.  Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Maja Pantic,et al.  Hierarchical On-line Appearance-Based Tracking for 3D head pose, eyebrows, lips, eyelids and irises , 2013, Image Vis. Comput..

[3]  Andrew W. Fitzgibbon,et al.  Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences , 2016, ACM Trans. Graph..

[4]  Christian Theobalt,et al.  Reconstructing detailed dynamic face geometry from monocular video , 2013, ACM Trans. Graph..

[5]  Hongdong Li,et al.  A simple prior-free method for non-rigid structure-from-motion factorization , 2012, CVPR.

[6]  Vincent Lepetit,et al.  Training a Feedback Loop for Hand Pose Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Dinesh K. Pai,et al.  Thin skin elastodynamics , 2013, ACM Trans. Graph..

[8]  S. Baron-Cohen,et al.  The "Reading the Mind in the Eyes" Test revised version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism. , 2001, Journal of child psychology and psychiatry, and allied disciplines.

[9]  Andrea Tagliasacchi,et al.  Robust Articulated-ICP for Real-Time Hand Tracking , 2015 .

[10]  R. Sibson,et al.  A brief description of natural neighbor interpolation , 1981 .

[11]  Timothy F. Cootes,et al.  On representing edge structure for model matching , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[12]  Vincent Lepetit,et al.  Closed-Form Solution to Non-rigid 3D Surface Registration , 2008, ECCV.

[13]  Yangang Wang,et al.  Online modeling for realtime facial animation , 2013, ACM Trans. Graph..

[14]  Dinesh K. Pai,et al.  Interactive gaze driven animation of the eye region , 2016, Web3D.

[15]  Derek Bradley,et al.  Detailed spatio-temporal reconstruction of eyelids , 2015, ACM Trans. Graph..

[16]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[17]  Alessio Del Bue,et al.  A factorization approach to structure from motion with shape priors , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Peter V. Gehler,et al.  Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image , 2016, ECCV.

[19]  Adrien Bartoli,et al.  Monocular Template-Based 3D Reconstruction of Extensible Surfaces with Local Linear Elasticity , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Aleix M. Martínez,et al.  Computing Smooth Time Trajectories for Camera and Deformable Shape in Structure from Motion with Occlusion , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Jing Xiao,et al.  Meticulously detailed eye region model and its application to analysis of facial images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Adrien Bartoli,et al.  On template-based reconstruction from a single view: Analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.