Symbol Spotting on Digital Architectural Floor Plans Using a Deep Learning-based Framework

This papers focuses on symbol spotting on real-world digital architectural floor plans with a deep learning (DL)-based framework. Traditional on-the-fly symbol spotting methods are unable to address the semantic challenge of graphical notation variability, i.e. low intra-class symbol similarity, an issue that is particularly important in architectural floor plan analysis. The presence of occlusion and clutter, characteristic of real-world plans, along with a varying graphical symbol complexity from almost trivial to highly complex, also pose challenges to existing spotting methods. In this paper, we address all of the above issues by leveraging recent advances in DL and adapting an object detection framework based on the You-Only-Look-Once (YOLO) architecture. We propose a training strategy based on tiles, avoiding many issues particular to DL-based object detection networks related to the relative small size of symbols compared to entire floor plans, aspect ratios, and data augmentation. Experiments on real-world floor plans demonstrate that our method successfully detects architectural symbols with low intra-class similarity and of variable graphical complexity, even in the presence of heavy occlusion and clutter. Additional experiments on the public SESYD dataset confirm that our proposed approach can deal with various degradation and noise levels and outperforms other symbol spotting methods.

[1]  Thomas M. Breuel,et al.  On the Use of Geometric Matching for Both: Isolated Symbol Recognition and Symbol Spotting , 2011, GREC.

[2]  Jean-Yves Ramel,et al.  Spotting Symbols in Line Drawing Images Using Graph Representations , 2007, GREC.

[3]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[4]  Umapada Pal,et al.  Near Convex Region Adjacency Graph and Approximate Neighborhood String Matching for Symbol Spotting in Graphical Documents , 2013, 2013 12th International Conference on Document Analysis and Recognition.

[5]  Salvatore Tabbone,et al.  A Symbol Spotting Approach Based on the Vector Model and a Visual Vocabulary , 2009, 2009 10th International Conference on Document Analysis and Recognition.

[6]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[7]  K. C. Santosh,et al.  Document Image Analysis: Current Trends and Challenges in Graphics Recognition , 2018 .

[8]  Sergio Escalera,et al.  Circular Blurred Shape Model for Multiclass Symbol Recognition , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[9]  Laurent Wendling,et al.  Matching of graphical symbols in line-drawing images using angular signature information , 2003, Document Analysis and Recognition.

[10]  Thomas M. Breuel,et al.  Statistical Grouping for Segmenting Symbols Parts from Line Drawings, with Application to Symbol Spotting , 2011, 2011 International Conference on Document Analysis and Recognition.

[11]  Jacques Labiche,et al.  Symbol Spotting using Full Visibility Graph Representation , 2007 .

[12]  Laurent Wendling,et al.  Integrating vocabulary clustering with spatial relations for symbol recognition , 2013, International Journal on Document Analysis and Recognition (IJDAR).

[13]  Simone Marinai,et al.  Object recognition in floor plans by graphs of white connected components , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[14]  Ernest Valveny,et al.  Report on the Symbol Recognition and Spotting Contest , 2011, GREC.

[15]  Yves Lecourtier,et al.  Symbol Detection Using Region Adjacency Graphs and Integer Linear Programming , 2009, 2009 10th International Conference on Document Analysis and Recognition.

[16]  Jean-Yves Ramel,et al.  Subgraph Spotting through Explicit Graph Embedding: An Application to Content Spotting in Graphic Document Images , 2011, 2011 International Conference on Document Analysis and Recognition.

[17]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Josep Lladós,et al.  Relational indexing of vectorial primitives for symbol spotting in line-drawing images , 2010, Pattern Recognit. Lett..

[19]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[20]  Josep Lladós,et al.  A Region-Based Hashing Approach for Symbol Spotting in Technical Documents , 2007, GREC.

[21]  Josep Lladós,et al.  Hierarchical Graph Representation for Symbol Spotting in Graphical Document Images , 2012, SSPR/SPR.

[22]  Nibaran Das,et al.  GSD-Net: Compact Network for Pixel-Level Graphical Symbol Detection , 2019, 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW).

[23]  Umapada Pal,et al.  Shallow Neural Network Model for Hand-Drawn Symbol Recognition in Multi-Writer Scenario , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[24]  Salvatore Tabbone,et al.  Symbol Descriptor Based on Shape Context and Vector Model of Information Retrieval , 2008, 2008 The Eighth IAPR International Workshop on Document Analysis Systems.

[25]  Jean-Yves Ramel,et al.  A structural representation for understanding line-drawing images , 2000, International Journal on Document Analysis and Recognition.

[26]  Sergio Escalera,et al.  Blurred Shape Model for binary and grey-level symbol recognition , 2009, Pattern Recognit. Lett..

[27]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[28]  Salvatore Tabbone,et al.  Symbol spotting for technical documents: An efficient template-matching approach , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[29]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  K. C. SANTOSH,et al.  Bor: Bag-of-Relations for symbol Retrieval , 2014, Int. J. Pattern Recognit. Artif. Intell..

[31]  Josep Lladós,et al.  Hierarchical Plausibility-Graphs for Symbol Spotting in Graphical Documents , 2013, GREC.

[32]  Yves Lecourtier,et al.  An integer linear program for substitution-tolerant subgraph isomorphism and its use for symbol spotting in technical drawings , 2012, Pattern Recognit..

[33]  Josep Lladós,et al.  A performance evaluation protocol for symbol spotting systems in terms of recognition and location indices , 2009, International Journal on Document Analysis and Recognition (IJDAR).

[34]  Umapada Pal,et al.  A symbol spotting approach in graphical documents by hashing serialized graphs , 2013, Pattern Recognit..

[35]  Laurent Wendling,et al.  Symbol recognition using spatial relations , 2012, Pattern Recognit. Lett..

[36]  Alexandra Branzan Albu,et al.  Symbol spotting for architectural drawings: state-of-the-art and new industry-driven developments , 2019, IPSJ Transactions on Computer Vision and Applications.

[37]  Simone Marinai,et al.  Object Detection in Floor Plan Images , 2018, ANNPR.

[38]  Shreya Goyal,et al.  BRIDGE: Building Plan Repository for Image Description Generation, and Evaluation , 2019, 2019 International Conference on Document Analysis and Recognition (ICDAR).

[39]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Su Yang Symbol Recognition via Statistical Integration of Pixel-Level Constraint Histograms: A New Descriptor , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[44]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[45]  Alicia Fornés,et al.  Graph-Based Deep Learning for Graphics Classification , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).