Panoramic Structure from Motion via Geometric Relationship Detection

This paper addresses the problem of Structure from Motion (SfM) for indoor panoramic image streams, extremely challenging even for the state-of-the-art due to the lack of textures and minimal parallax. The key idea is the fusion of single-view and multi-view reconstruction techniques via geometric relationship detection (e.g., detecting 2D lines as coplanar in 3D). Rough geometry suffices to perform such detection, and our approach utilizes rough surface normal estimates from an image-to-normal deep network to discover geometric relationships among lines. The detected relationships provide exact geometric constraints in our line-based linear SfM formulation. A constrained linear least squares is used to reconstruct a 3D model and camera motions, followed by the bundle adjustment. We have validated our algorithm on challenging datasets, outperforming various state-of-the-art reconstruction techniques.

[1]  Richard Szeliski,et al.  A Multi-stage Linear Approach to Structure from Motion , 2010, ECCV Workshops.

[2]  Yinda Zhang,et al.  PanoContext: A Whole-Room 3D Context Model for Panoramic Scene Understanding , 2014, ECCV.

[3]  Andrew Owens,et al.  SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels , 2013, 2013 IEEE International Conference on Computer Vision.

[4]  Roberto Manduchi,et al.  Planar Structures from Line Correspondences in a Manhattan World , 2014, ACCV.

[5]  Svetlana Lazebnik,et al.  Learning Informative Edge Maps for Indoor Scene Layout Prediction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[7]  Yael Pritch,et al.  Megastereo: Constructing High-Resolution Stereo Panoramas , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[9]  João Manuel R. S. Tavares,et al.  A new approach for merging edge line segments , 1995 .

[10]  Jaishanker K. Pillai,et al.  Manhattan Junction Catalogue for Spatial Reasoning of Indoor Scenes , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Abhinav Gupta,et al.  Designing deep networks for surface normal estimation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Derek Hoiem,et al.  Recovering the spatial layout of cluttered rooms , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[13]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[14]  Frank Dellaert,et al.  Line-Based Structure from Motion for Urban Environments , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[15]  Kyoung Mu Lee,et al.  Visual SLAM with Line and Corner Features , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[16]  Harry Shum,et al.  Stereo reconstruction from multiperspective panoramas , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Rafael Grompone von Gioi,et al.  LSD: A Fast Line Segment Detector with a False Detection Control , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Paul H. J. Kelly,et al.  Dense planar SLAM , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[19]  Hui Zhang,et al.  Efficient 3D Room Shape Recovery from a Single Panorama , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Noah Snavely.  Photo Tourism : Exploring image collections in 3D , 2006 .

[21]  Andrew Owens,et al.  Discrete-continuous optimization for large-scale structure from motion , 2011, CVPR 2011.

[22]  Matthew Brand,et al.  Lifting 3D Manhattan Lines from a Single Image , 2013, 2013 IEEE International Conference on Computer Vision.

[23]  In-So Kweon,et al.  High-Quality Depth from Uncalibrated Small Motion Clip , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Jianxiong Xiao,et al.  Image-based street-side city modeling , 2009, ACM Trans. Graph..

[25]  Ahmed M. Elgammal,et al.  Line-based relative pose estimation , 2011, CVPR 2011.

[26]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[27]  Guosheng Lin,et al.  Deep convolutional neural fields for depth estimation from a single image , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Ian D. Reid,et al.  A Dynamic Programming Approach to Reconstructing Building Interiors , 2010, ECCV.

[29]  Alexei A. Efros,et al.  Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics , 2010, ECCV.

[30]  Luc Van Gool,et al.  Joint vanishing point extraction and tracking , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  David J. Kriegman,et al.  Structure and Motion from Line Segments in Multiple Images , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Fisher Yu,et al.  3D Reconstruction from Accidental Motion , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.