Shape-From-Silhouette Across Time Part I: Theory and Algorithms

Shape-From-Silhouette (SFS) is a shape reconstruction method which constructs a 3D shape estimate of an object using silhouette images of the object. The output of a SFS algorithm is known as the Visual Hull (VH). Traditionally SFS is either performed on static objects, or separately at each time instant in the case of videos of moving objects. In this paper we develop a theory of performing SFS across time: estimating the shape of a dynamic object (with unknown motion) by combining all of the silhouette images of the object over time. We first introduce a one dimensional element called a Bounding Edge to represent the Visual Hull. We then show that aligning two Visual Hulls using just their silhouettes is in general ambiguous and derive the geometric constraints (in terms of Bounding Edges) that govern the alignment. To break the alignment ambiguity, we combine stereo information with silhouette information and derive a Temporal SFS algorithm which consists of two steps: (1) estimate the motion of the objects over time (Visual Hull Alignment) and (2) combine the silhouette information using the estimated motion (Visual Hull Refinement). The algorithm is first developed for rigid objects and then extended to articulated objects. In the Part II of this paper we apply our temporal SFS algorithm to two human-related applications: (1) the acquisition of detailed human kinematic models and (2) marker-less motion tracking.

[1]  Takeo Kanade,et al.  A Multiple-Baseline Stereo , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Aldo Laurentini,et al.  How Far 3D Shapes Can Be Understood from 2D Silhouettes , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Takeo Kanade,et al.  A factorization method for affine structure from line correspondences , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  Mark D. Wheeler,et al.  Automatic Modeling and Localization for Object Recognition , 1996 .

[5]  Takeo Kanade,et al.  A subspace approach to layer extraction , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[6]  Anil K. Jain.  Fundamentals of Digital Image Processing , 2018, Control of Color Imaging Systems.

[7]  Takeo Kanade,et al.  A Paraperspective Factorization Method for Shape and Motion Recovery , 1994, ECCV.

[8]  Ioannis A. Kakadiaris,et al.  Three-Dimensional Human Body Model Acquisition from Multiple Views , 1998, International Journal of Computer Vision.

[9]  G. Cheung.  Visual Hull Construction, Alignment and Refinement Across Time , 2001 .

[10]  Jake K. Aggarwal,et al.  Rectangular parallelepiped coding: A volumetric representation of three-dimensional objects , 1986, IEEE J. Robotics Autom..

[11]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[12]  Narendra Ahuja,et al.  Generating Octrees from Object Silhouettes in Orthographic Views , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[14]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[15]  Takeo Kanade,et al.  Visual hull alignment and refinement across time: a 3D reconstruction algorithm combining shape-from-silhouette with stereo , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[16]  Michael Potmesil.  Generating octree models of 3D objects from their silhouettes in a sequence of images , 1987, Comput. Vis. Graph. Image Process..

[17]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[18]  Olivier D. Faugeras,et al.  3D articulated models and multi-view tracking with silhouettes , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[19]  Aldo Laurentini.  The visual hull of curved objects , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[20]  Mohammed Yeasin,et al.  Automatic Acquisition and Initialization of Kinematic Models , 2001 .

[21]  Roberto Cipolla,et al.  Structure and motion from silhouettes , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[22]  Aldo Laurentini,et al.  The visual hull: a new tool for contour-based image understanding , 1991 .

[23]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[24]  Hiroshi Noborio,et al.  Construction of the Octree Approximating a Three-Dimensional Object by Using Multiple Views , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Arun K. Pujari,et al.  Volume intersection with optimal set of directions , 1991, Pattern Recognit. Lett..

[26]  Saied Moezzi,et al.  Virtual View Generation for 3D Digital Video , 1997, IEEE Multim..

[27]  Narendra Ahuja,et al.  Structure and Motion Estimation from Dynamic Silhouettes under Perspective Projection , 1995, Proceedings of IEEE International Conference on Computer Vision.

[28]  Paulo R. S. Mendonça,et al.  Head Model Acquisition from Silhouettes , 2001, IWVF.

[29]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Harpreet S. Sawhney,et al.  Compact Representations of Videos Through Dominant and Multiple Motion Estimation , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Katsushi Ikeuchi,et al.  Simultaneous 2D images and 3D geometric model registration for texture mapping utilizing reflectance attribute , 2002 .

[32]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[33]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[34]  Jean Ponce,et al.  On computing exact visual hulls of solids bounded by smooth surfaces , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[35]  Paulo R. S. Mendonça,et al.  Camera Pose Estimation and Reconstruction from Image Profiles under Circular Motion , 2000, ECCV.

[36]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[37]  Jake K. Aggarwal,et al.  Articulated and elastic non-rigid motion: a review , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[38]  David J. Kriegman,et al.  Structure and motion of curved 3D objects from monocular silhouettes , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[39]  J. Ponce,et al.  Towards structure and motion estimation from dynamic silhouettes , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[40]  Wojciech Matusik,et al.  Creating and Rendering Image-Based Visual Hulls , 1999 .

[41]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[42]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Richard Szeliski,et al.  Stereo Matching with Transparency and Matting , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[44]  Zhengyou Zhang,et al.  Iterative point matching for registration of free-form curves and surfaces , 1994, International Journal of Computer Vision.

[45]  Tal Hassner,et al.  What Does the Scene Look Like from a Scene Point? , 2002, ECCV.

[46]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[47]  Ioannis A. Kakadiaris,et al.  3D human body model acquisition from multiple views , 1995, Proceedings of IEEE International Conference on Computer Vision.

[48]  Richard Szeliski,et al.  Rapid octree construction from image sequences , 1993 .

[49]  Simon Baker,et al.  Visual hull construction, alignment and refinement for human kinematic modeling, motion tracking and rendering , 2003 .

[50]  Bruce G. Baumgart,et al.  Geometric modeling for computer vision. , 1974 .

[51]  Jake K. Aggarwal,et al.  TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE , 2008 .

[52]  Paulo R. S. Mendonça,et al.  Epipolar geometry from profiles under circular motion , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Mohammed Yeasin,et al.  Automatic acquisition and initialization of articulated models , 2003, Machine Vision and Applications.