Reliable Estimation of Deterioration Levels via Late Fusion Using Multi-View Distress Images for Practical Inspection

This paper presents reliable estimation of deterioration levels via late fusion using multi-view distress images for practical inspection. The proposed method simultaneously solves the following two problems that are necessary to support the practical inspection. Since maintenance of infrastructures requires a high level of safety and reliability, this paper proposes a neural network that can generate an attention map from distress images and text data acquired during the inspection. Thus, deterioration level estimation with high interpretability can be realized. In addition, since multi-view distress images are taken for single distress during the actual inspection, it is necessary to estimate the final result from these images. Therefore, the proposed method integrates estimation results obtained from the multi-view images via the late fusion and can derive an appropriate result considering all the images. To the best of our knowledge, no method has been proposed to solve these problems simultaneously, and this achievement is the biggest contribution of this paper. In this paper, we confirm the effectiveness of the proposed method by conducting experiments using data acquired during the actual inspection.

[1]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Japan,et al.  Transfer Learning-based Road Damage Detection for Multiple Countries , 2020, ArXiv.

[3]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Miki Haseyama,et al.  Convolutional sparse coding‐based deep random vector functional link network for distress classification of road structures , 2019, Comput. Aided Civ. Infrastructure Eng..

[5]  Yang Zhang,et al.  A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations , 2018, ICML.

[6]  Nagul Cooharojananone,et al.  Bridge Sub Structure Defect Inspection Assistance by using Deep Learning , 2019, 2019 IEEE 10th International Conference on Awareness Science and Technology (iCAST).

[7]  Jeff A. Bilmes,et al.  Deep Canonical Correlation Analysis , 2013, ICML.

[8]  Miki Haseyama,et al.  Deterioration level estimation via neural network maximizing category-based ordinally supervised multi-view canonical correlation , 2020, Multimedia Tools and Applications.

[9]  Miki Haseyama,et al.  Correlation-Aware Attention Branch Network Using Multi-Modal Data For Deterioration Level Estimation Of Infrastructures , 2021, ICIP 2021.

[10]  Takato Yasuno,et al.  Per-pixel Classification Rebar Exposures in Bridge Eye-inspection , 2020, ArXiv.

[11]  Louis-Philippe Morency,et al.  Multimodal Machine Learning: A Survey and Taxonomy , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  M. Haseyama,et al.  Distress Image Retrieval for Infrastructure Maintenance via Self-Trained Deep Metric Learning Using Experts’ Knowledge , 2021, IEEE Access.

[13]  Hironobu Fujiyoshi,et al.  Embedding Human Knowledge in Deep Neural Network via Attention Map , 2019, VISIGRAPP.

[14]  Miki Haseyama,et al.  Distress classification of class-imbalanced inspection data via correlation-maximizing weighted extreme learning machine , 2018, Adv. Eng. Informatics.

[15]  Bappaditya Mandal,et al.  Interleaved Deep Artifacts-Aware Attention Mechanism for Concrete Structural Defect Classification , 2021, IEEE Transactions on Image Processing.

[16]  Wei Li,et al.  Concrete defects inspection and 3D mapping using CityFlyer quadrotor robot , 2020, IEEE/CAA Journal of Automatica Sinica.

[17]  Miki Haseyama,et al.  Estimation of Deterioration Levels of Transmission Towers via Deep Learning Maximizing Canonical Correlation Between Heterogeneous Features , 2018, IEEE Journal of Selected Topics in Signal Processing.

[18]  Yoshihide Sekimoto,et al.  Generative adversarial network for road damage detection , 2020, Comput. Aided Civ. Infrastructure Eng..

[19]  Miki Haseyama,et al.  Distress Classification of Road Structures via Adaptive Bayesian Network Model Selection , 2017 .

[20]  Vineeth N. Balasubramanian,et al.  Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[21]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Wei Zhang,et al.  Unified Vision‐Based Methodology for Simultaneous Concrete Defect Detection and Geolocalization , 2018, Comput. Aided Civ. Infrastructure Eng..

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  Miki Haseyama,et al.  Distress Level Classification of Road Infrastructures via CNN Generating Attention Map , 2020, 2020 IEEE 2nd Global Conference on Life Sciences and Technologies (LifeTech).

[25]  Peter Söderholm,et al.  Data Analysis for Condition‐Based Railway Infrastructure Maintenance , 2015, Qual. Reliab. Eng. Int..

[26]  Sang-Kyun Woo,et al.  Development of the Corrosion Deterioration Inspection Tool for Transmission Tower Members , 2016 .

[27]  Hironobu Fujiyoshi,et al.  Attention Branch Network: Learning of Attention Mechanism for Visual Explanation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Niladri B. Puhan,et al.  Multi-Deformation Aware Attention Learning for Concrete Structural Defect Classification , 2021, IEEE Transactions on Circuits and Systems for Video Technology.

[29]  Ralph R. Martin,et al.  Attention mechanisms in computer vision: A survey , 2021, Computational Visual Media.

[30]  Miki Haseyama,et al.  Automatic estimation of deterioration level on transmission towers via deep extreme learning machine based on local receptive field , 2017, 2017 IEEE International Conference on Image Processing (ICIP).