Key Issues in Modeling of Complex 3D Structures from Video Sequences

Construction of three-dimensional structures from video sequences has wide applications for intelligent video analysis. This paper summarizes the key issues of the theory and surveys the recent advances in the state of the art. Reconstruction of a scene object from video sequences often takes the basic principle of structure from motion with an uncalibrated camera. This paper lists the typical strategies and summarizes the typical solutions or algorithms for modeling of complex three-dimensional structures. Open difficult problems are also suggested for further study.

[1]  Guanghui Wang,et al.  Stratification Approach for 3-D Euclidean Reconstruction of Nonrigid Objects From Uncalibrated Image Sequences , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  John Oliensis,et al.  A Critique of Structure-from-Motion Algorithms , 2000, Comput. Vis. Image Underst..

[3]  Daphna Weinshall,et al.  Dual Computation of Projective Shape and Camera Positions from Multiple Images , 1998, International Journal of Computer Vision.

[4]  Anders Heyden,et al.  Affine Structure and Motion from Points, Lines and Conics , 1999, International Journal of Computer Vision.

[5]  Adrien Bartoli,et al.  Algorithms for Batch Matrix Factorization with Application to Structure-from-Motion , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Richard Szeliski,et al.  A Multi-stage Linear Approach to Structure from Motion , 2010, ECCV Workshops.

[7]  Amnon Shashua,et al.  Trajectory Triangulation: 3D Reconstruction of Moving Points from a Monocular Image Sequence , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Allen R. Hanson,et al.  Decision Making and Uncertainty Management in a 3D Reconstruction System , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Michael Brady,et al.  Practical Structure and Motion from Stereo When Motion is Unconstrained , 2000, International Journal of Computer Vision.

[10]  Seth J. Teller,et al.  Wide-Area Egomotion Estimation from Known 3D Structure , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Richard Szeliski,et al.  Reconstructing Rome , 2010, Computer.

[12]  Richard Szeliski,et al.  Towards Internet-scale multi-view stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[13]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Hujun Bao,et al.  Consistent Depth Maps Recovery from a Video Sequence , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Richard I. Hartley,et al.  Critical Configurations for Projective Reconstruction from Multiple Views , 2005, International Journal of Computer Vision.

[16]  Arnold W. M. Smeulders,et al.  Stages as Models of Scene Geometry , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Richard Szeliski,et al.  Recovering 3D Shape and Motion from Image Streams Using Nonlinear Least Squares , 1994, J. Vis. Commun. Image Represent..

[18]  Jan-Michael Frahm,et al.  Detailed Real-Time Urban 3D Reconstruction from Video , 2007, International Journal of Computer Vision.

[19]  Anders Heyden,et al.  Reconstruction of General Curves, Using Factorization and Bundle Adjustment , 2004, International Journal of Computer Vision.

[20]  Nassir Navab,et al.  Relative Affine Structure: Canonical Model for 3D From 2D Geometry and Applications , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Anil K. Jain,et al.  3D Model-Based Face Recognition in Video , 2007, ICB.

[22]  Steven M. Seitz,et al.  Scene Summarization for Online Image Collections , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[23]  K. Takaya,et al.  Depth Measurement and 3D Metric Reconstruction from Two Uncalibrated Stereo Images , 2007, 2007 Canadian Conference on Electrical and Computer Engineering.

[24]  Marc Pollefeys,et al.  Interactive 3D architectural modeling from unordered photo collections , 2008, SIGGRAPH 2008.

[25]  Ian D. Reid,et al.  Single View Metrology , 2000, International Journal of Computer Vision.

[26]  Li Zhang,et al.  Model evolution: An incremental approach to non-rigid structure from motion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  Adrian Hilton,et al.  Scene modelling from sparse 3D data , 2005, Image Vis. Comput..

[28]  Richard Szeliski,et al.  Structure from motion for scenes with large duplicate structures , 2011, CVPR 2011.

[29]  Andrea Fusiello,et al.  Stabilizing 3D modeling with geometric constraints propagation , 2009, Comput. Vis. Image Underst..

[30]  Xin Liu,et al.  Shape from silhouette outlines using an adaptive dandelion model , 2007, Comput. Vis. Image Underst..

[31]  Kiriakos N. Kutulakos,et al.  Non-rigid structure from locally-rigid motion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Adrien Bartoli,et al.  Constrained Structure and Motion From Multiple Uncalibrated Views of a Piecewise Planar Scene , 2003, International Journal of Computer Vision.

[33]  Éric Marchand,et al.  Active Vision for Complete Scene Reconstruction and Exploration , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Yiannis Aloimonos,et al.  Structure from Motion: Beyond the Epipolar Constraint , 2000, International Journal of Computer Vision.

[35]  Richard Szeliski,et al.  Skeletal graphs for efficient structure from motion , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Christophe Collewet,et al.  Visual Servoing Based on Structure From Controlled Motion or on Robust Statistics , 2008, IEEE Transactions on Robotics.

[37]  David J. Kriegman,et al.  Structure and Motion from Line Segments in Multiple Images , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Takeo Kanade,et al.  A Paraperspective Factorization Method for Shape and Motion Recovery , 1994, ECCV.

[39]  Minas E. Spetsakis,et al.  Structure from motion using line correspondences , 1990, International Journal of Computer Vision.

[40]  Michal Havlena,et al.  Efficient Structure from Motion by Graph Optimization , 2010, ECCV.

[41]  José M. F. Moura,et al.  A fast algorithm for rigid structure from image sequences , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[42]  Richard Szeliski,et al.  Building Rome in a day , 2009, ICCV.

[43]  Chen Liang,et al.  3D reconstruction using silhouettes from unordered viewpoints , 2010, Image Vis. Comput..

[44]  Hujun Bao,et al.  Efficient Non-consecutive Feature Tracking for Structure-from-Motion , 2010, ECCV.

[45]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[46]  Michel Dhome,et al.  Generic and real-time structure from motion using local bundle adjustment , 2009, Image Vis. Comput..

[47]  Gérard G. Medioni,et al.  Model-Assisted 3D Face Reconstruction from Video , 2007, AMFG.

[48]  Takashi Matsuyama,et al.  Topology matching for 3D video compression , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  Gong Rubin,et al.  3D Structure from a Single Calibrated View Using Distance Constraints , 2003 .

[50]  J. Heel.  Direct Estimation of Structure and Motion from Multiple Frames , 1990 .

[51]  Silvio Savarese,et al.  Semantic structure from motion , 2011, CVPR 2011.

[52]  Adrien Bartoli,et al.  Structure-from-motion using lines: Representation, triangulation, and bundle adjustment , 2005, Comput. Vis. Image Underst..

[53]  Reinhard Koch,et al.  Visual Modeling with a Hand-Held Camera , 2004, International Journal of Computer Vision.

[54]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[55]  Michael Goesele,et al.  Scene Reconstruction and Visualization From Community Photo Collections , 2010, Proceedings of the IEEE.

[56]  Stefan Carlsson,et al.  Uncalibrated Motion Capture Exploiting Articulated Structure Constraints , 2004, International Journal of Computer Vision.

[57]  Guanghui Wang,et al.  What can we learn about the scene structure from three orthogonal vanishing points in images , 2009, Pattern Recognit. Lett..

[58]  Alex Pentland,et al.  3D structure from 2D motion , 1999, IEEE Signal Process. Mag..

[59]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[60]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[61]  Andrew Owens,et al.  Discrete-continuous optimization for large-scale structure from motion , 2011, CVPR 2011.

[62]  Pietro Parodi,et al.  3D Shape Reconstruction by Using Vanishing Points , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[63]  Andrew Zisserman,et al.  Multi-view Matching for Unordered Image Sets, or "How Do I Organize My Holiday Snaps?" , 2002, ECCV.

[64]  Richard Szeliski,et al.  Modeling the World from Internet Photo Collections , 2008, International Journal of Computer Vision.

[65]  Richard I. Hartley,et al.  Euclidean Reconstruction from Uncalibrated Views , 1993, Applications of Invariance in Computer Vision.

[66]  H. C. Longuet-Higgins,et al.  A computer algorithm for reconstructing a scene from two projections , 1981, Nature.

[67]  John Oliensis,et al.  Dealing with Noise in Multiframe Structure from Motion , 1999, Comput. Vis. Image Underst..

[68]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[69]  Fernando Jaureguizar,et al.  Photorealistic 3D reconstruction from handheld cameras , 2005, Machine Vision and Applications.

[70]  Denis Laurendeau,et al.  3D surface modeling from curves , 2004, Image Vis. Comput..

[71]  Andrew Owens,et al.  Discrete-continuous optimization for large-scale structure from motion , 2011, CVPR.

[72]  Olivier D. Faugeras,et al.  3-D Reconstruction of Urban Scenes from Image Sequences , 1998, Comput. Vis. Image Underst..

[73]  Long Quan,et al.  Resampling Structure from Motion , 2010, ECCV.

[74]  Jiro Katto,et al.  Structure Recovery with Multiple Cameras from Scaled Orthographic and Perspective Views , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[75]  Yang Song,et al.  Tour the world: Building a web-scale landmark recognition engine , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[76]  Luc Van Gool,et al.  3D Urban Scene Modeling Integrating Recognition and Reconstruction , 2008, International Journal of Computer Vision.

[77]  Maxime Lhuillier.  Toward Flexible 3D Modeling using a Catadioptric Camera , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[78]  Narendra Ahuja,et al.  Structure and Motion Estimation from Dynamic Silhouettes under Perspective Projection , 1995, Proceedings of IEEE International Conference on Computer Vision.

[79]  Amnon Shashua,et al.  Multiple View Geometry of General Algebraic Curves , 2004, International Journal of Computer Vision.

[80]  Yücel Yemez,et al.  A volumetric fusion technique for surface reconstruction from silhouettes and range data , 2007, Comput. Vis. Image Underst..

[81]  Paul E. Debevec,et al.  Digitizing the Parthenon: Estimating Surface Reflectance Properties of a Complex Scene under Captured Natural Illumination , 2004, VMV.

[82]  Richard Szeliski,et al.  Reconstructing building interiors from images , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[83]  David W. Murray,et al.  A unifying framework for structure and motion recovery from image sequences , 1995, Proceedings of IEEE International Conference on Computer Vision.

[84]  Adrien Bartoli,et al.  A Batch Algorithm for Implicit Non-rigid Shape and Motion Recovery , 2006, WDV.

[85]  Anders Heyden,et al.  Reconstructing Open Surfaces from Image Data , 2006, International Journal of Computer Vision.

[86]  Richard I. Hartley,et al.  A linear method for reconstruction from lines and points , 1995, Proceedings of IEEE International Conference on Computer Vision.

[87]  João Paulo Costeira,et al.  Estimating 3D shape from degenerate sequences with missing data , 2009, Comput. Vis. Image Underst..

[88]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[89]  Andrew W. Fitzgibbon,et al.  Automatic Camera Recovery for Closed or Open Image Sequences , 1998, ECCV.

[90]  Peter F. Sturm,et al.  A Factorization Based Algorithm for Multi-Image Projective Structure and Motion , 1996, ECCV.

[91]  Fadi Dornaika,et al.  Stereo geometry from 3D ego-motion streams , 2003, IEEE Trans. Syst. Man Cybern. Part B.