Hand-3d-Studio: A New Multi-View System for 3d Hand Reconstruction

This paper proposes a new system named as Hand-3D-Studio to capture the 3D hand pose and shape information. Our system includes 15 synchronized DSLR cameras, which can acquire high quality multi-view 4K resolution color images in a circular manner. We then introduce a 2D hand keypoints guided iterative pixel growth matching strategy for 3D reconstruction, where the 2D keypoints are obtained via convolution neural network. We find that the pre-detected 2D hand keypoints can greatly remove the matching noise, and thus improve the performance of reconstruction. After that, a non-rigid iterative closest points algorithm is performed to drive a template hand to fit the point clouds and register all the hand meshes. As a consequence, we captured more than 20K high quality hand color images, annotated 2D hand key-points, 3D point cloud as well as the registered hand meshes (>200). All the data are public on the website http://www.yangangwang.com for future research.

[1]  Andrew W. Fitzgibbon,et al.  Accurate, Robust, and Flexible Real-time Hand Tracking , 2015, CHI.

[2]  Thomas Brox,et al.  Learning to Estimate 3D Hand Pose from Single RGB Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[3]  Hao Li,et al.  Global Correspondence Optimization for Non‐Rigid Registration of Depth Scans , 2008, Comput. Graph. Forum.

[4]  Tae-Kyun Kim,et al.  Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Thomas Brox,et al.  FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[7]  Sergio Orts,et al.  Large-scale Multiview 3D Hand Pose Dataset , 2017, Image Vis. Comput..

[8]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[9]  Scott Schaefer,et al.  Image deformation using moving least squares , 2006, ACM Trans. Graph..

[10]  Luc Van Gool,et al.  Motion Capture of Hands in Action Using Discriminative Salient Points , 2012, ECCV.

[11]  Christian Theobalt,et al.  Real-Time Hand Tracking Under Occlusion from an Egocentric RGB-D Sensor , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[12]  Yangang Wang,et al.  SRHandNet: Real-Time 2D Hand Pose Estimation With Simultaneous Region Localization , 2019, IEEE Transactions on Image Processing.

[13]  Yebin Liu,et al.  Mask-Pose Cascaded CNN for 2D Hand Pose Estimation From Single Color Image , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[14]  Christian Theobalt,et al.  GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Marc Pollefeys,et al.  A multiple-camera system calibration toolbox using a feature descriptor-based calibration pattern , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.