4D Capture Using Visibility Information of Multiple Projector Camera System

In this paper, we propose a method with multiple cameras and projectors for 4D capture of moving objects. The issues of previous 4D capture systems are that the number of cameras are limited, and the number of images is very large to capture the sequence at high frame rate. We propose a multiple projector-camera system to tackle this problem. One of the issues of multi-view stereo is to determine visibility of cameras for each point of the surface. While estimating the scene geometry and its visibility is a chicken-and-egg problem for passive multi-view stereo, it was solved by, for example, iterative approach conducting the estimation of visibility and the reconstruction of the scene geometry repeatedly. With our method, since visibility problem is independently solved by using the projected pattern, shapes are recovered efficiently without considering visibility problem. Further, the visibility information is not only used for multi-view stereo reconstruction, but also for merging 3D shapes to eliminate inconsistency between devices. The efficiency of the proposed method is tested in the experiments, proving the merged mesh is suitable for 4Dreconstruction.

[1]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[2]  Adrian Hilton,et al.  The Multiple-Camera 3-D Production Studio , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[3]  Adrian Hilton,et al.  Reliable Surface Reconstructiuon from Multiple Range Images , 1996, ECCV.

[4]  Peter Liepa,et al.  Filling Holes in Meshes , 2003, Symposium on Geometry Processing.

[5]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Adrian Hilton,et al.  Volumetric Stereo with Silhouette and Feature Constraints , 2006, BMVC.

[7]  Bruno Raffin,et al.  Virtualization gate , 2009, SIGGRAPH '09.

[8]  Olivier D. Faugeras,et al.  Variational principles, surface evolution, PDEs, level set methods, and the stereo problem , 1998, IEEE Trans. Image Process..

[9]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Paolo Cignoni,et al.  Metro: Measuring Error on Simplified Surfaces , 1998, Comput. Graph. Forum.

[11]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[12]  Victor S. Lempitsky,et al.  Oriented Visibility for Multiview Reconstruction , 2006, ECCV.

[13]  Emmanuel Prados,et al.  Gradient Flows for Optimizing Triangular Mesh-based Surfaces: Applications to 3D Reconstruction Problems Dealing with Visibility , 2011, International Journal of Computer Vision.

[14]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[15]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[16]  Takeo Kanade,et al.  A real time system for robust 3D voxel reconstruction of human motions , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[17]  Jovan Popović,et al.  Dynamic shape capture using multi-view photometric stereo , 2009, SIGGRAPH 2009.

[18]  Qionghai Dai,et al.  Fusing Multiview and Photometric Stereo for 3D Reconstruction under Uncalibrated Illumination , 2011, IEEE Transactions on Visualization and Computer Graphics.

[19]  Nozomu Kasuya,et al.  One-Shot Entire Shape Scanning by Utilizing Multiple Projector-Camera Constraints of Grid Patterns , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[20]  Roberto Cipolla,et al.  Probabilistic visibility for multi-view stereo , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Jean-Philippe Pons,et al.  High Accuracy and Visibility-Consistent Dense Multiview Stereo , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  O. Faugeras,et al.  Variational principles, surface evolution, PDE's, level set methods and the stereo problem , 1998, 5th IEEE EMBS International Summer School on Biomedical Imaging, 2002..

[23]  Yasushi Yagi,et al.  One-shot Entire Shape Acquisition Method Using Multiple Projectors and Cameras , 2010, 2010 Fourth Pacific-Rim Symposium on Image and Video Technology.

[24]  Katsushi Ikeuchi,et al.  Adaptively merging large-scale range data with reflectance properties , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Xiaojun Wu,et al.  Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video , 2004, Comput. Vis. Image Underst..

[26]  Pedro F. Felzenszwalb,et al.  Efficient belief propagation for early vision , 2004, CVPR 2004.

[27]  Daniel Cremers,et al.  Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[29]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[30]  Tony DeRose,et al.  Mesh optimization , 1993, SIGGRAPH.

[31]  Yasushi Yagi,et al.  Grid-Based Active Stereo with Single-Colored Wave Pattern for Dense One-shot 3D Scan , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[32]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[33]  Denis Laurendeau,et al.  A General Surface Approach to the Integration of a Set of Range Views , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Yasushi Yagi,et al.  Entire Shape Acquisition Technique Using Multiple Projectors and Cameras with Parallel Pattern Projection , 2012, IPSJ Trans. Comput. Vis. Appl..