Vision-based autonomous bolt-looseness detection method for splice connections: Design, lab-scale evaluation, and field application

Abstract This study presents a novel autonomous vision-based bolt-looseness detection method for splice bolted connections. The method is sequentially designed with a Faster regional convolutional neural network-based bolt detector, an automatic distortion corrector, an adaptive bolt-angle estimator, and a bolt-looseness classifier. The robustness of the method is demonstrated by detecting loosened bolts in a lab-scale bolted joint under sharp capturing angles and different lighting conditions. Next, the method is applied to detect loosened bolts in a realistic joint of the Dragon Bridge in Danang, Vietnam. The bolt detector shows the training, validation, and testing accuracy of 98.85%, 97.48%, and 93%, respectively. Loosened bolts in the lab-scale and real-scale joints are well detected with precisely-estimated loosening severities, even for a sharp perspective angle. The method also shows a high level of adaptability with low-brightness images. Therefore, the method has great potentials for autonomous monitoring of in-situ bolted connections.

[1]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[3]  Jochen Moll,et al.  Temperature affected guided wave propagation in a composite plate complementing the Open Guided Waves Platform , 2019, Scientific Data.

[4]  Thanh-Canh Huynh,et al.  Bolt-loosening identification of bolt connections by vision image-based technique , 2016, SPIE Smart Structures and Materials + Nondestructive Evaluation and Health Monitoring.

[5]  Wongi S. Na.  Bolt loosening detection using impedance based non-destructive method and probabilistic neural network technique with minimal training data , 2021 .

[6]  Thierry Moreau,et al.  Leveraging the VTA-TVM Hardware-Software Stack for FPGA Acceleration of 8-bit ResNet-18 Inference , 2018, ReQuEST@ASPLOS.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Atsushi Ike,et al.  Speed-Up of Object Detection Neural Network with GPU , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[9]  Wensheng Su,et al.  Autonomous bolt loosening detection using deep learning , 2019, Structural Health Monitoring.

[10]  Grzegorz Świt,et al.  Dragon bridge - the world largest dragon-shaped (ARCH) steel bridge as element of smart city , 2016 .

[11]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[12]  Herbert Edelsbrunner,et al.  Weighted alpha shapes , 1992 .

[13]  Ting Liu,et al.  Recent advances in convolutional neural networks , 2015, Pattern Recognit..

[14]  Zhengyao Bai,et al.  On the Multi-scale Real-Time Object Detection Using ResNet , 2019, PRCV.

[15]  R. I. Zadoks,et al.  AN INVESTIGATION OF THE SELF-LOOSENING BEHAVIOR OF BOLTS UNDER TRANSVERSE VIBRATION , 1997 .

[16]  K. Maurya,et al.  Smart materials and electro-mechanical impedance technique: A review , 2020 .

[17]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[18]  Hoon Sohn,et al.  Overview of Piezoelectric Impedance-Based Health Monitoring and Path Forward , 2003 .

[19]  Hyung Jin Lim,et al.  Impedance based damage detection under varying temperature and loading conditions , 2011 .

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Chuan-Yu Chang,et al.  Practical Homography-based perspective correction method for License Plate Recognition , 2012, 2012 International Conference on Information Security and Intelligent Control.

[22]  Dongho Kang,et al.  Autonomous UAVs for Structural Health Monitoring Using Deep Learning and an Ultrasonic Beacon System with Geo‐Tagging , 2018, Comput. Aided Civ. Infrastructure Eng..

[23]  Taghi M. Khoshgoftaar,et al.  A survey of transfer learning , 2016, Journal of Big Data.

[24]  Miroslav Pástor,et al.  Modal Assurance Criterion , 2012 .

[25]  Niannian Wang,et al.  Bolt loosening angle detection technology using deep learning , 2018, Structural Control and Health Monitoring.

[26]  Xindong Wu,et al.  Object Detection With Deep Learning: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Rishi Gupta,et al.  Health Monitoring of Civil Structures with Integrated UAV and Image Processing System , 2015 .

[28]  Duzgun Agdas,et al.  Comparison of visual inspection and structural-health monitoring as bridge condition assessment methods , 2016 .

[29]  Oral Büyüköztürk,et al.  Autonomous Structural Visual Inspection Using Region‐Based Deep Learning for Detecting Multiple Damage Types , 2018, Comput. Aided Civ. Infrastructure Eng..

[30]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Young-Jin Cha,et al.  Vision-based detection of loosened bolts using the Hough transform and support vector machines , 2016 .

[32]  Young-Jin Cha,et al.  Fully automated vision-based loosened bolt detection using the Viola–Jones algorithm , 2019 .

[33]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[34]  Marcin Skoczylas,et al.  Faster R-CNN:an Approach to Real-Time Object Detection , 2018, 2018 International Conference and Exposition on Electrical And Power Engineering (EPE).

[35]  Gangbing Song,et al.  Design of a New Vision-Based Method for the Bolts Looseness Detection in Flange Connections , 2020, IEEE Transactions on Industrial Electronics.

[36]  Gangbing Song,et al.  Review of Bolted Connection Monitoring , 2013, Int. J. Distributed Sens. Networks.

[37]  Eugenio Culurciello,et al.  An Analysis of Deep Neural Network Models for Practical Applications , 2016, ArXiv.

[38]  Joseph L. Rose,et al.  Guided wave mode and frequency selection tips , 2014 .

[39]  Thanh-Canh Huynh,et al.  Quantification of temperature effect on impedance monitoring via PZT interface for prestressed tendon anchorage , 2017 .

[40]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[42]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[43]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[44]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[45]  Ranita Biswas,et al.  An Improved Canny Edge Detection Algorithm Based on Type-2 Fuzzy Sets , 2012 .

[46]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[47]  David G. Kirkpatrick,et al.  On the shape of a set of points in the plane , 1983, IEEE Trans. Inf. Theory.

[48]  Hyung-Jo Jung,et al.  Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing , 2019, Automation in Construction.

[49]  Jeong-Tae Kim,et al.  Vision-based technique for bolt-loosening detection in wind turbine tower , 2015 .

[50]  Jeong-Tae Kim,et al.  Bolt-Loosening Monitoring Framework Using an Image-Based Deep Learning and Graphical Model , 2020, Sensors.

[51]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  G. Dinger,et al.  Avoiding self-loosening failure of bolted joints with numerical assessment of local contact state , 2011 .

[54]  Jeong-Tae Kim,et al.  Preload Monitoring in Bolted Connection Using Piezoelectric-Based Smart Interface , 2018, Sensors.

[55]  Hoon Sohn,et al.  Integrated impedance and guided wave based damage detection , 2012 .

[56]  Paolo Napoletano,et al.  Anomaly Detection in Nanofibrous Materials by CNN-Based Self-Similarity , 2018, Sensors.