NeurVPS: Neural Vanishing Point Scanning via Conic Convolution

We present a simple yet effective end-to-end trainable deep network with geometry-inspired convolutional operators for detecting vanishing points in images. Traditional convolutional neural networks rely on aggregating edge features and do not have mechanisms to directly exploit the geometric properties of vanishing points as the intersections of parallel lines. In this work, we identify a canonical conic space in which the neural network can effectively compute the global geometric information of vanishing points locally, and we propose a novel operator named conic convolution that can be implemented as regular convolutions in this space. This new operator explicitly enforces feature extractions and aggregations along the structural lines and yet has the same number of parameters as the regular 2D convolution. Our extensive experiments on both synthetic and real-world datasets show that the proposed operator significantly improves the performance of vanishing point detection over traditional methods. The code and dataset have been made publicly available at https://github.com/zhou13/neurvps.

[1]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Zihan Zhou,et al.  Detecting Dominant Vanishing Points in Natural Scenes with Application to Composition-Sensitive Image Retrieval , 2016, IEEE Transactions on Multimedia.

[3]  Stéphane Mallat,et al.  Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  G. F. McLean,et al.  Vanishing Point Detection by Line Clustering , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Jake K. Aggarwal,et al.  Determining vanishing points from perspective images , 1984, Comput. Vis. Graph. Image Process..

[6]  Roberto Cipolla,et al.  Camera Calibration from Vanishing Points in Image of Architectural Scenes , 1999, BMVC.

[7]  Scott Workman,et al.  Horizon Lines in the Wild , 2016, BMVC.

[8]  Trevor Darrell,et al.  Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields , 2019, ArXiv.

[9]  Scott Workman,et al.  Detecting Vanishing Points Using Global Image Context in a Non-ManhattanWorld , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Pushmeet Kohli,et al.  Geometric Image Parsing in Man-Made Environments , 2010, International Journal of Computer Vision.

[11]  Ali Borji Vanishing point detection with convolutional neural networks , 2016, ArXiv.

[12]  Yi Ma,et al.  End-to-End Wireframe Parsing , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Alan L. Yuille,et al.  Manhattan World: compass direction from a single image by Bayesian inference , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[14]  Stéphane Mallat,et al.  Invariant Scattering Convolution Networks , 2012, IEEE transactions on pattern analysis and machine intelligence.

[15]  James H. Elder,et al.  Efficient Edge-Based Methods for Estimating Manhattan Frames in Urban Imagery , 2008, ECCV.

[16]  Frank Dellaert,et al.  Atlanta world: an expectation maximization framework for simultaneous low-level edge grouping and camera calibration in complex man-made environments , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[17]  Alexei A. Efros,et al.  Putting Objects in Perspective , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[18]  Álvaro González Measurement of Areas on a Sphere Using Fibonacci and Latitude–Longitude Lattices , 2009, 0912.4540.

[19]  Chen Feng,et al.  SEMI-AUTOMATIC 3D RECONSTRUCTION OF PIECEWISE PLANAR BUILDING MODELS FROM SINGLE IMAGE , 2010 .

[20]  Rafael Grompone von Gioi,et al.  LSD: A Fast Line Segment Detector with a False Detection Control , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Allan Hanbury,et al.  Robust camera self-calibration from monocular images of Manhattan worlds , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Wei Zhang,et al.  Video Compass , 2002, ECCV.

[23]  Jia Deng,et al.  Stacked Hourglass Networks for Human Pose Estimation , 2016, ECCV.

[24]  C. Coelho,et al.  Extraction of vanishing points from images of indoor and outdoor scenes , 1993, Image Vis. Comput..

[25]  Junmo Kim,et al.  Active Convolution: Learning the Shape of Convolution for Image Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Li-Yi Wei,et al.  Learning to Reconstruct 3D Manhattan Wireframes From a Single Image , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[27]  Max Welling,et al.  Spherical CNNs , 2018, ICLR.

[28]  P.V.C. Hough,et al.  Machine Analysis of Bubble Chamber Pictures , 1959 .

[29]  Leonidas J. Guibas,et al.  TextureNet: Consistent Local Parametrizations for Learning From High-Resolution Signals on Meshes , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[31]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[32]  Jean-Philippe Tardif,et al.  Non-iterative approach for fast and accurate vanishing point detection , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[33]  Long Quan,et al.  Determining perspective structures using hierarchical Hough transform , 1989, Pattern Recognit. Lett..

[34]  James F. O'Brien,et al.  Exposing photo manipulation with inconsistent reflections , 2012, TOGS.

[35]  Rafael Grompone von Gioi,et al.  Finding Vanishing Points via Point Alignments in Image Primal and Dual Domains , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Bodo Rosenhahn,et al.  Deep Learning for Vanishing Point Detection Using an Inverse Gnomonic Projection , 2017, GCPR.

[37]  Laurent Itti,et al.  DeepVP: Deep Learning for Vanishing Point Detection on 1 Million Street View Images , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[38]  Eric Maisel,et al.  Using vanishing points for camera calibration and coarse 3D reconstruction from a single image , 2000, The Visual Computer.

[39]  Michel Antunes,et al.  A Global Approach for the Detection of Vanishing Points and Mutually Orthogonal Vanishing Directions , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Xinbo Gao,et al.  Dominant vanishing point detection in the wild with application in composition analysis , 2018, Neurocomputing.

[41]  In So Kweon,et al.  VPGNet: Vanishing Point Guided Network for Lane and Road Marking Detection and Recognition , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[42]  Matthias Nießner,et al.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Stergios I. Roumeliotis,et al.  Optimal estimation of vanishing points in a Manhattan world , 2011, 2011 International Conference on Computer Vision.

[44]  Matthias Nießner,et al.  Spherical CNNs on Unstructured Grids , 2019, ICLR.

[45]  Pierre Vandergheynst,et al.  Geodesic Convolutional Neural Networks on Riemannian Manifolds , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[46]  Robert C. Bolles,et al.  A RANSAC-Based Approach to Model Fitting and Its Application to Finding Cylinders in Range Data , 1981, IJCAI.

[47]  Pascal Vasseur,et al.  Globally optimal line clustering and vanishing point estimation in Manhattan world , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[48]  Yi Li,et al.  Deformable Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).