Joint vanishing point extraction and tracking

We present a novel vanishing point (VP) detection and tracking algorithm for calibrated monocular image sequences. Previous VP detection and tracking methods usually assume known camera poses for all frames or detect and track separately. We advance the state-of-the-art by combining VP extraction on a Gaussian sphere with recent advances in multi-target tracking on probabilistic occupancy fields. The solution is obtained by solving a Linear Program (LP). This enables the joint detection and tracking of multiple VPs over sequences. Unlike existing works we do not need known camera poses, and at the same time avoid detecting and tracking in separate steps. We also propose an extension to enforce VP orthogonality. We augment an existing video dataset consisting of 48 monocular videos with multiple annotated VPs in 14448 frames for evaluation. Although the method is designed for unknown camera poses, it is also helpful in scenarios with known poses, since a multi-frame approach in VP detection helps to regularize in frames with weak VP line support.

[1]  Jake K. Aggarwal,et al.  Determining vanishing points from perspective images , 1984, Comput. Vis. Graph. Image Process..

[2]  Seth J. Teller,et al.  Automatic recovery of relative camera rotations for urban scenes , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[3]  Pushmeet Kohli,et al.  Geometric Image Parsing in Man-Made Environments , 2010, ECCV.

[4]  Marc Pollefeys,et al.  3-line RANSAC for orthogonal vanishing point detection , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[5]  Peyman Moghadam,et al.  Road direction detection based on vanishing-point tracking , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  John J. Leonard,et al.  A Mixture of Manhattan Frames: Beyond the Manhattan World , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Anthony Hoogs,et al.  A Minimum Error Vanishing Point Detection Approach for Uncalibrated Monocular Images of Man-Made Environments , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Allan Hanbury,et al.  Robust camera self-calibration from monocular images of Manhattan worlds , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Pascal Vasseur,et al.  Rotation estimation and vanishing point extraction by omnidirectional vision in urban environment , 2012, Int. J. Robotics Res..

[10]  Michel Antunes,et al.  A Global Approach for the Detection of Vanishing Points and Mutually Orthogonal Vanishing Directions , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Stephen T. Barnard,et al.  Interpreting Perspective Image , 1983, Artif. Intell..

[12]  Rafael Grompone von Gioi,et al.  Finding Vanishing Points via Point Alignments in Image Primal and Dual Domains , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  James H. Elder,et al.  Efficient Edge-Based Methods for Estimating Manhattan Frames in Urban Imagery , 2008, ECCV.

[14]  Roberto Manduchi,et al.  Planar Structures from Line Correspondences in a Manhattan World , 2014, ACCV.

[15]  Horst Wildenauer,et al.  Minimal Solution for Uncalibrated Absolute Pose Problem with a Known Vanishing Point , 2013, 2013 International Conference on 3D Vision.

[16]  R. Leconge,et al.  Tracking orthogonal vanishing points in video sequences for a reliable camera orientation in Manhattan World , 2012, 2012 5th International Congress on Image and Signal Processing.

[17]  B. Caprile,et al.  Using vanishing points for camera calibration , 1990, International Journal of Computer Vision.

[18]  Jaime López-Krahe,et al.  Contribution to the Determination of Vanishing Points Using Hough Transform , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  F. Dellaert,et al.  Atlanta world: an expectation maximization framework for simultaneous low-level edge grouping and camera calibration in complex man-made environments , 2004, CVPR 2004.

[20]  Carsten Rother.  A new approach to vanishing point detection in architectural environments , 2002, Image Vis. Comput..

[21]  Pascal Fua,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Multiple Object Tracking Using K-shortest Paths Optimization , 2022 .

[22]  Ramakant Nevatia,et al.  Global data association for multi-object tracking using network flows , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Ram Nevatia,et al.  Learning to associate: HybridBoosted multi-target tracker for crowded scene , 2009, CVPR.

[24]  Έλλη Πέτσα,et al.  An automatic approach for camera calibration from vanishing points , 2015 .

[25]  Andrew Owens,et al.  Discrete-continuous optimization for large-scale structure from motion , 2011, CVPR 2011.

[26]  Long Quan,et al.  Determining perspective structures using hierarchical Hough transform , 1989, Pattern Recognit. Lett..

[27]  Luc Van Gool,et al.  The Cascaded Hough Transform as Support for Grouping and Finding Vanishing Points and Lines , 1997, AFPAC.

[28]  Rafael Grompone von Gioi,et al.  LSD: a Line Segment Detector , 2012, Image Process. Line.

[29]  Luc Van Gool,et al.  Video Registration to SfM Models , 2014, ECCV.

[30]  Jean-Philippe Tardif,et al.  Non-iterative approach for fast and accurate vanishing point detection , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[31]  Wei Zhang,et al.  Video Compass , 2002, ECCV.

[32]  Stefan Maierhofer,et al.  Extracting vanishing points across multiple views , 2011, CVPR 2011.

[33]  Christopher Rasmussen,et al.  RoadCompass: following rural roads with vision + ladar using vanishing point tracking , 2008, Auton. Robots.

[34]  Rainer Stiefelhagen,et al.  Multiple Object Tracking Performance Metrics and Evaluation in a Smart Room Environment , 2006 .

[35]  Luc Van Gool,et al.  Discovery of Sets of Mutually Orthogonal Vanishing Points in Videos , 2015, 2015 IEEE Winter Applications and Computer Vision Workshops.