Discovery of Sets of Mutually Orthogonal Vanishing Points in Videos

While vanishing point (VP) estimation has received extensive attention, most approaches focus on static images or perform detection and tracking separately. In this paper, we focus on man-made environments and propose a novel method for detecting and tracking groups of mutually orthogonal vanishing points (MOVP), also known as Manhattan frames, jointly from monocular videos. The method is unique in that it is designed to enforce orthogonality in groups of VPs, temporal consistency of each individual MOVP, and orientation consistency of all putative MOVP. To this end, the method consists of three steps: 1) proposal of MOVP candidates by directly incorporating mutual orthogonality; 2) extracting consistent tracks of MOVPs by minimizing the flow cost over a network where nodes are putative MOVPs and edges are putative links across time; and 3) refinement of all MOVPs by enforcing consistency between lines, their identified vanishing directions and consistency of global camera orientation. The method is evaluated on six newly collected and annotated videos of urban scenes. Extensive experiments show that the method outperforms greedy MOVP tracking method considerably. In addition, we also test the method for camera orientation estimation and show that it obtains very promising results on a challenging street-view dataset.

[1]  Andrew V. Goldberg,et al.  An efficient implementation of a scaling minimum-cost flow algorithm , 1993, IPCO.

[2]  Rafael Grompone von Gioi,et al.  Finding Vanishing Points via Point Alignments in Image Primal and Dual Domains , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Long Quan,et al.  Determining perspective structures using hierarchical Hough transform , 1989, Pattern Recognit. Lett..

[4]  Du Q. Huynh,et al.  Metrics for 3D Rotations: Comparison and Analysis , 2009, Journal of Mathematical Imaging and Vision.

[5]  Frank Dellaert,et al.  Atlanta world: an expectation maximization framework for simultaneous low-level edge grouping and camera calibration in complex man-made environments , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[6]  Rafael Grompone von Gioi,et al.  LSD: a Line Segment Detector , 2012, Image Process. Line.

[7]  Luc Van Gool,et al.  Video Registration to SfM Models , 2014, ECCV.

[8]  Carsten Rother,et al.  A New Approach for Vanishing Point Detection in Architectural Environments , 2000, BMVC.

[9]  Michel Antunes,et al.  A Global Approach for the Detection of Vanishing Points and Mutually Orthogonal Vanishing Directions , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Christopher Rasmussen,et al.  RoadCompass: following rural roads with vision + ladar using vanishing point tracking , 2008, Auton. Robots.

[11]  John J. Leonard,et al.  A Mixture of Manhattan Frames: Beyond the Manhattan World , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Luc Van Gool,et al.  The Cascaded Hough Transform as Support for Grouping and Finding Vanishing Points and Lines , 1997, AFPAC.

[13]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[14]  James H. Elder,et al.  Efficient Edge-Based Methods for Estimating Manhattan Frames in Urban Imagery , 2008, ECCV.

[15]  Ramakant Nevatia,et al.  Learning to associate: HybridBoosted multi-target tracker for crowded scene , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Jaime López-Krahe,et al.  Contribution to the Determination of Vanishing Points Using Hough Transform , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  L. Grammatikopoulos,et al.  An automatic approach for camera calibration from vanishing points , 2007 .

[18]  Jean-Philippe Tardif,et al.  Non-iterative approach for fast and accurate vanishing point detection , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[19]  Anthony Hoogs,et al.  A Minimum Error Vanishing Point Detection Approach for Uncalibrated Monocular Images of Man-Made Environments , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Allan Hanbury,et al.  Robust camera self-calibration from monocular images of Manhattan worlds , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[22]  Roberto Manduchi,et al.  Planar Structures from Line Correspondences in a Manhattan World , 2014, ACCV.

[23]  Horst Wildenauer,et al.  Minimal Solution for Uncalibrated Absolute Pose Problem with a Known Vanishing Point , 2013, 2013 International Conference on 3D Vision.

[24]  F. Dellaert,et al.  Atlanta world: an expectation maximization framework for simultaneous low-level edge grouping and camera calibration in complex man-made environments , 2004, CVPR 2004.

[25]  Wei Zhang,et al.  Video Compass , 2002, ECCV.

[26]  Stefan Maierhofer,et al.  Extracting vanishing points across multiple views , 2011, CVPR 2011.

[27]  R. Leconge,et al.  Tracking orthogonal vanishing points in video sequences for a reliable camera orientation in Manhattan World , 2012, 2012 5th International Congress on Image and Signal Processing.

[28]  B. Caprile,et al.  Using vanishing points for camera calibration , 1990, International Journal of Computer Vision.

[29]  Andrew Owens,et al.  Discrete-continuous optimization for large-scale structure from motion , 2011, CVPR.

[30]  Ramakant Nevatia,et al.  Global data association for multi-object tracking using network flows , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Jake K. Aggarwal,et al.  Determining vanishing points from perspective images , 1984, Comput. Vis. Graph. Image Process..

[32]  Seth J. Teller,et al.  Automatic recovery of relative camera rotations for urban scenes , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[33]  Pushmeet Kohli,et al.  Geometric Image Parsing in Man-Made Environments , 2010, International Journal of Computer Vision.

[34]  Marc Pollefeys,et al.  3-line RANSAC for orthogonal vanishing point detection , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[35]  Peyman Moghadam,et al.  Road direction detection based on vanishing-point tracking , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.