Actor 3D reconstruction by a scene-based, visual hull guided, multi-stereovision framework

This paper proposes a novel framework to produce 3D, high-precision models of humans from multi-view capture. This method’s inputs are a visual hull and several sets of multi-baseline views. For each such view set, a surface is reconstructed with a multi-baseline stereovision method, then used to carve the visual hull. Carved visual hulls from different view sets are then fused pairwise to deliver the intended 3D model. The contributions of this paper are threefold: (i) the addition of visual hull guidance to a multi-baseline stereovision method, (ii) a carving solution to a visual hull from an interpolated and smooth stereovision surface, and (iii) a fusion solution to merge differently carved volumes differing in several areas. The paper shows that the proposed approach helps recovering a high quality carved volume, a 3D representation of the human to be modelled, that is precise even for small details and in concave areas subjected to occlusion.

[1]  Peter Eisert,et al.  3-D reconstruction of real-world objects using extended voxels , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[2]  Adrian Hilton,et al.  Multiple view reconstruction of people , 2004, Proceedings. 2nd International Symposium on 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004..

[3]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[4]  Takeo Kanade,et al.  A real time system for robust 3D voxel reconstruction of human motions , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[5]  Céline Loscos,et al.  Materiality maps: A novel scene-based framework for direct multi-view stereovision reconstruction , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[6]  Jake K. Aggarwal,et al.  Volume/surface octrees for the representation of three-dimensional objects , 1986, Comput. Vis. Graph. Image Process..

[7]  Pieter Abbeel,et al.  Range sensor and silhouette fusion for high-quality 3D Scanning , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[8]  Peng Song,et al.  Volumetric stereo and silhouette fusion for image-based modeling , 2010, The Visual Computer.

[9]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[10]  Céline Loscos,et al.  RECOVER3D: A Hybrid Multi-View System for 4D Reconstruction of Moving Actors , 2013 .

[11]  Richard Szeliski,et al.  Stereo Matching with Transparency and Matting , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[12]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[13]  Alan L. Yuille,et al.  Local, global, and multilevel stereo matching , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Kazuki Matsuda,et al.  Direct Shape Carving: Smooth 3D Points and Normals for Surface Reconstruction , 2012, IEICE Trans. Inf. Syst..

[15]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[16]  Hans-Peter Seidel,et al.  Combining stereo and visual hull information for on-line reconstruction and rendering of dynamic scenes , 2002, 2002 IEEE Workshop on Multimedia Signal Processing..

[17]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[18]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[19]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[20]  Yannick Rémion,et al.  An Occlusion Approach with Consistency Constraint for Multiscopic Depth Extraction , 2010, Int. J. Digit. Multim. Broadcast..