Multi-modal robotic visual-tactile localisation and detection of surface cracks

We present and validate a method to detect surface cracks with visual and tactile sensing. The proposed algorithm localises cracks in remote environments through videos/photos taken by an on-board robot camera. The identified areas of interest are then explored by a robot with a tactile sensor. Faster R-CNN object detection is used for identifying the location of potential cracks. Random forest classifier is used for tactile identification of the cracks to confirm their presences. Offline and online experiments to compare vision only and combined vision and tactile based crack detection are demonstrated. Two experiments are developed to test the efficiency of the multi-modal approach: online accuracy detection and time required to explore a surface and localise a crack. Exploring a cracked surface using combined visual and tactile modalities required four times less time than using the tactile modality only. The accuracy of detection was also improved with the combination of the two modalities. This approach may be implemented also in extreme environments since gamma radiation does not interfere with the sensing mechanism of fibre optic-based sensors.

[1]  T. Seguchi,et al.  Radiation resistance of nylon , 1983 .

[2]  Branko Glisic,et al.  Crack detection and characterization techniques—An overview , 2014 .

[3]  Qian Wang,et al.  DeepCrack: Learning Hierarchical Convolutional Features for Crack Detection , 2019, IEEE Transactions on Image Processing.

[4]  Mohammad R. Jahanshahi,et al.  An innovative methodology for detection and quantification of cracks through incorporation of depth perception , 2011, Machine Vision and Applications.

[5]  Kaspar Althoefer,et al.  Object classification using hybrid fiber optical force/proximity sensor , 2017, 2017 IEEE SENSORS.

[6]  Mohammad R. Jahanshahi,et al.  NB-CNN: Deep Learning-Based Crack Detection Using Convolutional Neural Network and Naïve Bayes Data Fusion , 2018, IEEE Transactions on Industrial Electronics.

[7]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[9]  Sumathi Poobal,et al.  Crack detection using image processing: A critical review and analysis , 2017, Alexandria Engineering Journal.

[10]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[11]  Peter Kenesei,et al.  Investigation of fatigue crack initiation from a non-metallic inclusion via high energy x-ray diffraction microscopy , 2017 .

[12]  Kaspar Althoefer,et al.  Automatic Fracture Characterization Using Tactile and Proximity Optical Sensing , 2020, Frontiers in Robotics and AI.

[13]  Shigeki Sugano,et al.  Design and Characterization of a Three-Axis Hall Effect-Based Soft Skin Sensor , 2016, Sensors.

[14]  Carlos Eduardo Pereira,et al.  Embedded Image Processing Systems for Automatic Recognition of Cracks using UAVs , 2015 .

[15]  Sergio Guadarrama,et al.  Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Kaspar Althoefer,et al.  Finger contact sensing and the application in dexterous hand manipulation , 2015, Autonomous Robots.

[17]  Yanyang Zi,et al.  An adaptive sparse deconvolution method for distinguishing the overlapping echoes of ultrasonic guided waves for pipeline crack inspection , 2017 .

[18]  Kaspar Althoefer,et al.  Implementing Tactile and Proximity Sensing for Crack Detection , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[19]  Hoang-Nam Nguyen,et al.  An Automatic Approach for Accurate Edge Detection of Concrete Crack Utilizing 2D Geometric Features of Crack , 2014, J. Signal Process. Syst..

[20]  Véronique Perdereau,et al.  Tactile sensing in dexterous robot hands - Review , 2015, Robotics Auton. Syst..

[21]  Robert J. Thomas,et al.  Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete , 2018, Construction and Building Materials.

[22]  Francis Berghmans,et al.  Radiation hardness of fiber optic sensors for monitoring and remote handling applications in nuclear environments , 1999, Other Conferences.

[23]  Ashutosh Bagchi,et al.  Image-based retrieval of concrete crack properties for bridge inspection , 2014 .

[24]  Ã.F. Ãzgenel,et al.  Performance Comparison of Pretrained Convolutional Neural Networks on Crack Detection in Buildings , 2018 .