Efficient Fourier-Based Approach for Detecting Orientations and Occlusions in Epipolar Plane Images for 3D Scene Modeling

This paper presents a Fourier-based approach for automatically constructing a 3D panoramic model of a natural scene from a video sequence. The video sequences could be captured by an unstabilized camera mounted on a moving platform on a common road surface. As the input of the algorithms, “seamles” panoramic view images (PVIs) and epipolar plane images (EPIs) are generated after image stabilization if the camera is unstabilized. A novel panoramic EPI analysis method is proposed that combines the advantages of both PVIs and EPIs efficiently in three important steps: locus orientation detection in the Fourier frequency domain, motion boundary localization in the spatio-temporal domain, and occlusion/resolution recovery only at motion boundaries. The Fourier energy-based approaches in literature were usually for low-level local motion analysis and are therefore not accurate for 3D reconstruction and are also computationally expensive. Our panoramic EPI analysis approach is both accurate and efficient for 3D reconstruction. Examples of layered panoramic representations for large-scale 3D scenes from real world video sequences are given.

[1]  Richard Szeliski,et al.  Stereo reconstruction from multiperspective panoramas , 2004 .

[2]  Richard Szeliski,et al.  Systems and Experiment Paper: Construction of Panoramic Image Mosaics with Global and Local Alignment , 2000, International Journal of Computer Vision.

[3]  Kenneth Turkowski,et al.  Creating image-based VR using a self-calibrating fisheye lens , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  Harry Shum,et al.  Stereo reconstruction from multiperspective panoramas , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[5]  Saburo Tsuji,et al.  Panoramic representation for route recognition by a mobile robot , 1992, International Journal of Computer Vision.

[6]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[7]  Richard Szeliski,et al.  Construction of Panoramic Image Mosaics with Global and Local Alignment , 2001 .

[8]  Michael J. Black,et al.  Estimating Optical Flow in Segmented Images Using Variable-Order Parametric Models With Local Deformations , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  David J. Heeger,et al.  Optical flow from spatialtemporal filters , 1987 .

[10]  Seth Teller,et al.  Automatic Extraction of Textured Vertical Facades from Pose Imagery , 1998 .

[11]  Kristin J. Dana,et al.  Real-time scene stabilization and mosaic construction , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[12]  Xueyin Lin,et al.  Constructing 3D natural scene from video sequences with vibrated motions , 1998, Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180).

[13]  Shmuel Peleg,et al.  Mosaicing on Adaptive Manifolds , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Robert C. Bolles,et al.  Generalizing Epipolar-Plane Image Analysis on the spatiotemporal surface , 2004, International Journal of Computer Vision.

[15]  Xueyin Lin,et al.  Panoramic EPI generation and analysis of video from a moving platform with vibration , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[16]  Harpreet S. Sawhney,et al.  Compact Representations of Videos Through Dominant and Multiple Motion Estimation , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  R. Chellappa,et al.  Fast 3D stabilization and mosaic construction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Paul Rademacher,et al.  Multiple-center-of-projection images , 1998, SIGGRAPH.

[19]  Robert T. Collins,et al.  A space-sweep approach to true multi-image matching , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[20]  Sourabh A. Niyogi,et al.  Detecting kinetic occlusion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[21]  Allen R. Hanson,et al.  Parallel-perspective stereo mosaics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[22]  S. Suzuki,et al.  Construction of Panoramic Image with Distance Information , 2006, 2006 SICE-ICASE International Joint Conference.

[23]  David J. Fleet,et al.  Motion feature detection using steerable flow fields , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[24]  Allen R. Hanson,et al.  Generalized parallel-perspective stereo mosaics from airborne video , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Andrew Zisserman,et al.  Automatic reconstruction of piecewise planar models from multiple views , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[26]  Hiroshi Ishiguro,et al.  Omni-directional stereo for making global map , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[27]  Saburo Tsuji,et al.  Generating Dynamic Projection Images for Scene Representation and Understanding , 1998, Comput. Vis. Image Underst..

[28]  Charles R. Dyer,et al.  Long-range spatiotemporal motion understanding using spatiotemporal flow curves , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[29]  Harpreet S. Sawhney,et al.  VideoBrush/sup TM/: experiences with consumer video mosaicing , 1998, Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201).

[30]  Edward H. Adelson,et al.  The Design and Use of Steerable Filters , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Mei Han,et al.  Interactive construction of 3D models from panoramic mosaics , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[32]  Robert C. Bolles,et al.  Epipolar-plane image analysis: An approach to determining structure from motion , 1987, International Journal of Computer Vision.

[33]  David W. Murray.  Recovering Range Using Virtual Multicamera Stereo , 1995, Comput. Vis. Image Underst..

[34]  Richard Szeliski,et al.  A multi-view approach to motion and stereo , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[35]  Shmuel Peleg,et al.  Stereo panorama with a single camera , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[36]  Shmuel Peleg,et al.  Panoramic mosaics by manifold projection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[37]  Bernd Jähne,et al.  Digital Image Processing: Concepts, Algorithms, and Scientific Applications , 1991 .

[38]  Olivier D. Faugeras,et al.  3-D Reconstruction of Urban Scenes from Image Sequences , 1998, Comput. Vis. Image Underst..

[39]  J BlackMichael,et al.  Estimating Optical Flow in Segmented Images Using Variable-Order Parametric Models With Local Deformations , 1996 .

[40]  Zhu Zhigang.  Full View Spatio-Temporal Visual Navigation Imaging , Modeling and Representation of Real Scenes , .

[41]  Allen R. Hanson,et al.  3D LAMP: a new layered panoramic representation , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[42]  Mohan M. Trivedi,et al.  High-Speed Extraction of 3D Structure of Selectable Quality Using a Translating Camera , 1996, Comput. Vis. Image Underst..

[43]  Allen R. Hanson,et al.  The Ascender System: Automated Site Modeling from Multiple Aerial Images , 1998, Comput. Vis. Image Underst..

[44]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[45]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.

[46]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[47]  Avideh Zakhor,et al.  Constructing a Multivalued Representation for View Synthesis , 2001, International Journal of Computer Vision.

[48]  E H Adelson,et al.  Spatiotemporal energy models for the perception of motion. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[49]  Avideh Zakhor,et al.  View generation for three-dimensional scenes from video sequences , 1997, IEEE Trans. Image Process..

[50]  Shree K. Nayar,et al.  360/spl times/360 mosaics , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[51]  Harry Shum,et al.  Omnivergent Stereo , 2004, International Journal of Computer Vision.

[52]  Yael Pritch,et al.  Omnistereo: Panoramic Stereo Imaging , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  Harry Shum,et al.  Stereo reconstruction from multiperspective panoramas , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[55]  Richard Szeliski,et al.  A layered approach to stereo reconstruction , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).