Vision‐based automated bridge component recognition with high‐level scene consistency

This research investigates vision‐based automated bridge component recognition, which is critical for automating visual inspection of bridges during initial response after earthquakes. Semantic segmentation algorithms with up to 45 convolutional layers are applied to recognize bridge components from images of complex scenes. One of the challenges in such scenarios is to get the recognition results consistent with high‐level scene structure using limited amount of training data. To impose the high‐level scene consistency, this research combines 10‐class scene classification and 5‐class bridge component classification. Three approaches are investigated to combine scene classification results into bridge component classification: (a) naïve configuration, (b) parallel configuration, and (c) sequential configuration of classifiers. The proposed approaches, sequential configuration in particular, are demonstrated to be effective in recognizing bridge components in complex scenes, showing less than 1% of accuracy loss from the naïve/parallel configuration for bridge images, and less than 1% false positives for the nonbridge images.

[1]  Thomas J. Hacker,et al.  Postevent Reconnaissance Image Documentation Using Automated Classification , 2019, Journal of Performance of Constructed Facilities.

[2]  Khalid M. Mosalam,et al.  Deep Transfer Learning for Image‐Based Structural Damage Recognition , 2018, Comput. Aided Civ. Infrastructure Eng..

[3]  Yicheng Li,et al.  A Fast Detection Method via Region‐Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects , 2018, Comput. Aided Civ. Infrastructure Eng..

[4]  Ruodan Lu,et al.  Detection of Structural Components in Point Clouds of Existing RC Bridges , 2018, Comput. Aided Civ. Infrastructure Eng..

[5]  B. F. Spencer,et al.  Vision-based Automated Bridge Component Recognition Integrated With High-level Scene Understanding , 2018, ArXiv.

[6]  Billie F. Spencer,et al.  Automated Vision-based Bridge Component Extraction Using Multiscale Convolutional Neural Networks , 2018, ArXiv.

[7]  B. F. Spencer,et al.  Vision-based Structural Inspection using Multiscale Deep Convolutional Neural Networks , 2018, ArXiv.

[8]  Yang Liu,et al.  Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network , 2017, Comput. Aided Civ. Infrastructure Eng..

[9]  Luc Van Gool,et al.  Fast Scene Understanding for Autonomous Driving , 2017, ArXiv.

[10]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[11]  Yuxin Peng,et al.  Object-Part Attention Model for Fine-Grained Image Classification , 2017, IEEE Transactions on Image Processing.

[12]  Shirley J. Dyke,et al.  Autonomous image localization for visual inspection of civil infrastructure , 2017 .

[13]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[14]  Joost van de Weijer,et al.  Hierarchical part detection with deep neural networks , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[15]  Antonio M. López,et al.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Wolfram Burgard,et al.  Deep learning for human part discovery in images , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[19]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Chul Min Yeum,et al.  Vision‐Based Automated Crack Detection for Bridge Inspection , 2015, Comput. Aided Civ. Infrastructure Eng..

[23]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Rob Fergus,et al.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[27]  Trevor Darrell,et al.  Fully convolutional networks for semantic segmentation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[29]  Markus König,et al.  Achievements and Challenges in Machine Vision-Based Inspection of Large Concrete Structures , 2014 .

[30]  Jonathan Krause,et al.  3D Object Representations for Fine-Grained Categorization , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[31]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[33]  Alex Greer,et al.  Earthquake Preparedness and Response: Comparison of the United States and Japan , 2012 .

[34]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Scenes and Its Applications , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Ioannis Brilakis,et al.  Concrete Column Recognition in Images and Videos , 2010, J. Comput. Civ. Eng..

[36]  Stephen Gould,et al.  Decomposing a scene into geometric and semantically consistent regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[37]  Roberto Cipolla,et al.  Semantic object classes in video: A high-definition ground truth database , 2009, Pattern Recognit. Lett..

[38]  Roberto Cipolla,et al.  Segmentation and Recognition Using Structure from Motion Point Clouds , 2008, ECCV.

[39]  Ram Nevatia,et al.  Detection and Segmentation of Multiple, Partially Occluded Objects by Grouping, Merging, Assigning Part Detection Responses , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[41]  Cordelia Schmid,et al.  Human Detection Based on a Probabilistic Assembly of Robust Part Detectors , 2004, ECCV.

[42]  Anders Krogh,et al.  A Simple Weight Decay Can Improve Generalization , 1991, NIPS.

[43]  S. J. Dyke,et al.  Big visual data analytics for damage classification in civil engineering , 2015 .

[44]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..