PPGNet: Learning Point-Pair Graph for Line Segment Detection

In this paper, we present a novel framework to detect line segments in man-made environments. Specifically, we propose to describe junctions, line segments and relationships between them with a simple graph, which is more structured and informative than end-point representation used in existing line segment detection methods. In order to extract a line segment graph from an image, we further introduce the PPGNet, a convolutional neural network that directly infers a graph from an image. We evaluate our method on published benchmarks including York Urban and Wireframe datasets. The results demonstrate that our method achieves satisfactory performance and generalizes well on all the benchmarks. The source code of our work is available at https://github.com/svip-lab/PPGNet.

[1]  Azriel Rosenfeld,et al.  Angle Detection on Digital Curves , 1973, IEEE Transactions on Computers.

[2]  Marsha Jo Hannah,et al.  Computer matching of areas in stereo images. , 1974 .

[3]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[4]  Pietro Parodi,et al.  3D Shape Reconstruction by Using Vanishing Points , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Cordelia Schmid,et al.  Automatic line matching across views , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Farzin Mokhtarian,et al.  Robust Image Corner Detection Through Curvature Scale Space , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Laxmi Parida,et al.  Junctions: Detection, Classification, and Reconstruction , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Jiri Matas,et al.  Robust Detection of Lines Using the Progressive Probabilistic Hough Transform , 2000, Comput. Vis. Image Underst..

[9]  Miguel Cazorla,et al.  Two Bayesian methods for junction classification , 2003, IEEE Trans. Image Process..

[10]  Yoshihisa Shinagawa,et al.  Accurate and robust line segment extraction by analyzing distribution around peaks in Hough space , 2003, Comput. Vis. Image Underst..

[11]  Frédéric Cao,et al.  Good continuations in digital image level lines , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[12]  Jitendra Malik,et al.  Learning to detect natural image boundaries using local brightness, color, and texture cues , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Josh H. McDermott,et al.  Psychophysics with junctions in real images. , 2010, Perception.

[14]  Honglak Lee,et al.  Automatic Single-Image 3d Reconstructions of Indoor Manhattan World Scenes , 2007, ISRR.

[15]  Eric D. Sinzinger,et al.  A model-based approach to junction detection using radial energy , 2008, Pattern Recognit..

[16]  Jitendra Malik,et al.  Using contours to detect and localize junctions in natural images , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Guojun Lu,et al.  An Improved Curvature Scale-Space Corner Detector and a Robust Corner Matching Approach for Transformed Image Identification , 2008, IEEE Transactions on Image Processing.

[18]  James H. Elder,et al.  Efficient Edge-Based Methods for Estimating Manhattan Frames in Urban Imagery , 2008, ECCV.

[19]  Rafael Grompone von Gioi,et al.  LSD: A Fast Line Segment Detector with a False Detection Control , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Ahmed M. Elgammal,et al.  Line-based relative pose estimation , 2011, CVPR 2011.

[21]  Ian D. Reid,et al.  Manhattan scene understanding using monocular, stereo, and 3D features , 2011, 2011 International Conference on Computer Vision.

[22]  Narciso García,et al.  Line segment detection using weighted mean shift procedures on a 2D slice sampling strategy , 2011, Pattern Analysis and Applications.

[23]  Cuneyt Akinlar,et al.  EDLines: A real-time line segment detector with a false detection control , 2011, Pattern Recognit. Lett..

[24]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[25]  Jaishanker K. Pillai,et al.  Manhattan Junction Catalogue for Spatial Reasoning of Indoor Scenes , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Julie Delon,et al.  Accurate Junction Detection and Characterization in Natural Images , 2013, International Journal of Computer Vision.

[27]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[28]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Reinhard Koch,et al.  Structure and motion from line correspondences: Representation, projection, initialization and sparse bundle adjustment , 2014, J. Vis. Commun. Image Represent..

[30]  Yan Wang,et al.  DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[33]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[34]  Bok-Suk Shin,et al.  Accurate and Robust Line Segment Extraction Using Minimum Entropy With Hough Transform , 2015, IEEE Transactions on Image Processing.

[35]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[36]  Li Li,et al.  CannyLines: A parameter-free line segment detector , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[37]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  David Windridge,et al.  A generalisable framework for saliency-based line segment detection , 2015, Pattern Recognit..

[39]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[41]  Pascal Monasse,et al.  Robust and Accurate Line- and/or Point-Based Pose Estimation without Manhattan Assumptions , 2016, ECCV.

[42]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Zhiao Huang,et al.  Associative Embedding: End-to-End Learning for Joint Detection and Grouping , 2016, NIPS.

[44]  Tomasz Malisiewicz,et al.  RoomNet: End-to-End Room Layout Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[45]  Bolei Zhou,et al.  Scene Parsing through ADE20K Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Danfei Xu,et al.  Scene Graph Generation by Iterative Message Passing , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Jia Deng,et al.  Pixels to Graphs by Associative Embedding , 2017, NIPS.

[49]  Li Zhang,et al.  Physics Inspired Optimization on Semantic Transfer Features: An Alternative Method for Room Layout Estimation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  James H. Elder,et al.  MCMLSD: A Dynamic Programming Approach to Line Segment Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[52]  Horst Bischof,et al.  Efficient 3D scene abstraction using line segments , 2017, Comput. Vis. Image Underst..

[53]  Bolei Zhou,et al.  Semantic Understanding of Scenes Through the ADE20K Dataset , 2016, International Journal of Computer Vision.

[54]  Kaiming He,et al.  Group Normalization , 2018, ECCV.

[55]  Yuning Jiang,et al.  Unified Perceptual Parsing for Scene Understanding , 2018, ECCV.

[56]  Brian Coltin,et al.  Indoor RGB-D Compass from a Single Line and Plane , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[57]  Derek Hoiem,et al.  LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[58]  Kun Huang,et al.  Learning to Parse Wireframes in Images of Man-Made Environments , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[59]  Shi Jin,et al.  Automatic 3D Indoor Scene Modeling from Single Panorama , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[60]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.