Visual hull alignment and refinement across time: a 3D reconstruction algorithm combining shape-from-silhouette with stereo

Visual hull (VH) construction from silhouette images is a popular method of shape estimation. The method, also known as shape-from-silhouette (SFS), is used in many applications such as non-invasive 3D model acquisition, obstacle avoidance, and more recently human motion tracking and analysis. One of the limitations of SFS, however, is that the approximated shape can be very coarse when there are only a few cameras. In this paper, we propose an algorithm to improve the shape approximation by combining multiple silhouette images captured across time. The improvement is achieved by first estimating the rigid motion between the visual hulls formed at different time instants (visual hull alignment) and then combining them (visual hull refinement) to get a tighter bound on the object's shape. Our algorithm first constructs a representation of the VHs called the bounding edge representation. Utilizing a fundamental property of visual hulls, which states that each bounding edge must touch the object at at least one point, we use multi-view stereo to extract points called colored surface points (CSP) on the surface of the object. These CSPs are then used in a 3D image alignment algorithm to find the 6 DOF rigid motion between two visual hulls. Once the rigid motion across time is known, all of the silhouette images are treated as being captured at the same time instant and the shape of the object is refined. We validate our algorithm on both synthetic and real data and compare it with space carving.

[1]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[2]  Paulo R. S. Mendonça,et al.  Head Model Acquisition from Silhouettes , 2001, IWVF.

[3]  William H. Press,et al.  Numerical recipes in C , 2002 .

[4]  Aldo Laurentini.  The visual hull of curved objects , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[5]  Richard Szeliski,et al.  Rapid octree construction from image sequences , 1993 .

[6]  Richard Szeliski,et al.  Stereo Matching with Transparency and Matting , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[7]  Bruce G. Baumgart,et al.  Geometric modeling for computer vision. , 1974 .

[8]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Tal Hassner,et al.  What Does the Scene Look Like from a Scene Point? , 2002, ECCV.

[10]  G. Cheung.  Visual Hull Construction, Alignment and Refinement Across Time , 2001 .

[11]  David J. Kriegman,et al.  Structure and motion of curved 3D objects from monocular silhouettes , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[12]  Jake K. Aggarwal,et al.  Rectangular parallelepiped coding: A volumetric representation of three-dimensional objects , 1986, IEEE J. Robotics Autom..

[13]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[14]  Michael Potmesil.  Generating octree models of 3D objects from their silhouettes in a sequence of images , 1987, Comput. Vis. Graph. Image Process..

[15]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[16]  Jean Ponce,et al.  On computing exact visual hulls of solids bounded by smooth surfaces , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[17]  Paul A. Viola,et al.  Roxels: responsibility weighted 3D volume reconstruction , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[18]  Narendra Ahuja,et al.  Structure and Motion Estimation from Dynamic Silhouettes under Perspective Projection , 1995, Proceedings of IEEE International Conference on Computer Vision.

[19]  Roberto Cipolla,et al.  Structure and motion from silhouettes , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[20]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[21]  Wojciech Matusik,et al.  Creating and Rendering Image-Based Visual Hulls , 1999 .