Introduction of Deep Learning in Thermographic Monitoring of Cultural Heritage and Improvement by Automatic Thermogram Pre-Processing Algorithms

The monitoring of heritage objects is necessary due to their continuous deterioration over time. Therefore, the joint use of the most up-to-date inspection techniques with the most innovative data processing algorithms plays an important role to apply the required prevention and conservation tasks in each case study. InfraRed Thermography (IRT) is one of the most used Non-Destructive Testing (NDT) techniques in the cultural heritage field due to its advantages in the analysis of delicate objects (i.e., undisturbed, non-contact and fast inspection of large surfaces) and its continuous evolution in both the acquisition and the processing of the data acquired. Despite the good qualitative and quantitative results obtained so far, the lack of automation in the IRT data interpretation predominates, with few automatic analyses that are limited to specific conditions and the technology of the thermographic camera. Deep Learning (DL) is a data processor with a versatile solution for highly automated analysis. Then, this paper introduces the latest state-of-the-art DL model for instance segmentation, Mask Region-Convolution Neural Network (Mask R-CNN), for the automatic detection and segmentation of the position and area of different surface and subsurface defects, respectively, in two different artistic objects belonging to the same family: Marquetry. For that, active IRT experiments are applied to each marquetry. The thermal image sequences acquired are used as input dataset in the Mask R-CNN learning process. Previously, two automatic thermal image pre-processing algorithms based on thermal fundamentals are applied to the acquired data in order to improve the contrast between defective and sound areas. Good detection and segmentation results are obtained regarding state-of-the-art IRT data processing algorithms, which experience difficulty in identifying the deepest defects in the tests. In addition, the performance of the Mask R-CNN is improved by the prior application of the proposed pre-processing algorithms.

[1]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Xavier Maldague,et al.  Non-destructive Investigation of Paintings on Canvas by Continuous Wave Terahertz Imaging and Flash Thermography , 2017, Journal of Nondestructive Evaluation.

[3]  Pedro Arias,et al.  Thermographic methodologies used in infrastructure inspection: A review—Post-processing procedures , 2020 .

[4]  Dr. V. Ramesh.  A Review on Application of Deep Learning in Thermography , 2017 .

[5]  Lachgar Mohamed,et al.  An Overview of Gradient Descent Algorithm Optimization in Machine Learning: Application in the Ophthalmology Field , 2020 .

[6]  A. Bendada,et al.  Comparative Study of Active Thermography Techniques for the Nondestructive Evaluation of Honeycomb Structures , 2009 .

[7]  B. Marschner,et al.  The potential of active and passive infrared thermography for identifying dynamics of soil moisture and microbial activity at high spatial and temporal resolution , 2018, Geoderma.

[8]  John Chilton,et al.  Evaluation Tool For The Thermal Performance of Retrofitted Buildings Using An Integrated Approach of Deep Learning Artificial Neural Networks and Infrared Thermography , 2020 .

[9]  Rahmat Ali.  Deep learning- and infrared thermography-based subsurface damage detection in a steel bridge , 2019 .

[10]  Lorenzo Rosasco,et al.  Are Loss Functions All the Same? , 2004, Neural Computation.

[11]  Karin Ibenholt,et al.  Economic impacts of cultural heritage – Research and perspectives , 2009 .

[12]  Pedro Arias,et al.  Thermographic methodologies used in infrastructure inspection: A review—data acquisition procedures , 2020 .

[13]  Wai Lok Woo,et al.  Temporal and spatial deep learning network for infrared thermal defect detection , 2019, NDT & E International.

[14]  Bernt Schiele,et al.  Learning Non-maximum Suppression , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Koungsuk Kim,et al.  Quantitative determination of a subsurface defect of reference specimen by lock-in infrared thermography , 2008 .

[16]  Misha Denil,et al.  Predicting Parameters in Deep Learning , 2014 .

[17]  J. Balado,et al.  Thermal-based analysis for the automatic detection and characterization of thermal bridges in buildings , 2018 .

[18]  Amit P. Sheth,et al.  Machine learning for Internet of Things data analysis: A survey , 2017, Digit. Commun. Networks.

[19]  Stefano Sfarra,et al.  Photothermal coherence tomography for 3-D visualization and structural non-destructive imaging of a wood inlay , 2018, Infrared Physics & Technology.

[20]  İ. İnce,et al.  Deep learning-based weathering type recognition in historical stone monuments , 2020 .

[21]  X. Maldague,et al.  Evaluating quality of marquetries by applying active IR thermography and advanced signal processing , 2020, Journal of Thermal Analysis and Calorimetry.

[22]  Giovanni Maria Carlomagno,et al.  Application of infrared thermography and geophysical methods for defect detection in architectural structures , 2005 .

[23]  Pedro Arias,et al.  Development of Thermal Principles for the Automation of the Thermographic Monitoring of Cultural Heritage , 2020, Sensors.

[24]  S. Sfarra,et al.  Inspecting Marquetries at Different Wavelengths: The Preliminary Numerical Approach as Aid for a Wide-Range of Non-destructive Tests , 2017 .

[25]  Dimitrios I. Fotiadis,et al.  Machine learning applications in cancer prognosis and prediction , 2014, Computational and structural biotechnology journal.

[26]  Susana Lagüela-Lopez,et al.  IRT and GPR Techniques for Moisture Detection and Characterisation in Buildings , 2020, Sensors.

[27]  Xavier Maldague,et al.  Application of Deep Learning in Infrared Non-Destructive Testing , 2018 .

[28]  Georg Heigold,et al.  An empirical study of learning rates in deep neural networks for speech recognition , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[29]  Snehasis Mukherjee,et al.  Impact of Fully Connected Layers on Performance of Convolutional Neural Networks for Image Classification , 2019, Neurocomputing.

[30]  D Thickett,et al.  Using non-invasive non-destructive techniques to monitor cultural heritage objects , 2017 .

[31]  Stefano Sfarra,et al.  Automatic Detection and Delimitation of Internal Moisture in Mosaics from Thermographic Sequences. Experimental Tests , 2019 .

[32]  Roberto Pierdicca,et al.  Point Cloud Semantic Segmentation Using a Deep Learning Framework for Cultural Heritage , 2020, Remote. Sens..

[33]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[34]  Niall O' Mahony,et al.  Deep Learning vs. Traditional Computer Vision , 2019, CVC.

[35]  Oge Marques,et al.  On the use of variable stride in convolutional neural networks , 2020, Multimedia Tools and Applications.

[36]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[37]  Guodong Zhang,et al.  Three Mechanisms of Weight Decay Regularization , 2018, ICLR.

[38]  Xavier Maldague,et al.  Improving the detection of thermal bridges in buildings via on-site infrared thermography: The potentialities of innovative mathematical tools , 2019, Energy and Buildings.

[39]  Wei Li,et al.  Transferable Joint Attribute-Identity Deep Learning for Unsupervised Person Re-identification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Tara N. Sainath,et al.  Improving deep neural networks for LVCSR using rectified linear units and dropout , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[41]  Xavier Maldague,et al.  Active thermography testing and data analysis for the state of conservation of panel paintings , 2018 .

[42]  Aman Jantan,et al.  State-of-the-art in artificial neural network applications: A survey , 2018, Heliyon.

[43]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[44]  Stefano Sfarra,et al.  Infrared vision for artwork and cultural heritage NDE studies: principles and case studies , 2017 .

[45]  A. Restrepo-Girón,et al.  Optimized Gaussian model for non-uniform heating compensation in pulsed thermography. , 2020, Applied optics.

[46]  Yan Lu,et al.  Local Descriptors Optimized for Average Precision , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  Stefano Sfarra,et al.  Algorithms for the automatic detection and characterization of pathologies in heritage elements from thermographic images , 2019 .

[48]  T. Jayakumar,et al.  Infrared thermography for condition monitoring – A review , 2013 .

[49]  Hung Manh La,et al.  A Genetic Algorithm for Convolutional Network Structure Optimization for Concrete Crack Detection , 2018, 2018 IEEE Congress on Evolutionary Computation (CEC).

[50]  Jue Hu,et al.  Pattern Deep Region Learning for Crack Detection in Thermography Diagnosis System , 2018, Metals.

[51]  Pedro Arias,et al.  Autonomous thermography: towards the automatic detection and classification of building pathologies , 2018 .

[52]  Julio Molleda,et al.  Infrared Thermography for Temperature Measurement and Non-Destructive Testing , 2014, Sensors.

[53]  Clemente Ibarra-Castanedo,et al.  Automated defect classification in infrared thermography based on a neural network , 2019, NDT & E International.

[54]  David B. Lobell,et al.  Weakly Supervised Deep Learning for Segmentation of Remote Sensing Imagery , 2020, Remote. Sens..

[55]  Stefano Sfarra,et al.  Automatic detection of moistures in different construction materials from thermographic images , 2019, Journal of Thermal Analysis and Calorimetry.

[56]  Stefano Sfarra,et al.  Measuring the Water Content in Wood Using Step-Heating Thermography and Speckle Patterns-Preliminary Results , 2020, Sensors.

[57]  Xavier Maldague,et al.  Infrared image processing and data analysis , 2004 .

[58]  Luis Perez,et al.  The Effectiveness of Data Augmentation in Image Classification using Deep Learning , 2017, ArXiv.

[59]  Murat Yakar,et al.  Importance of digital close-range photogrammetry in documentation of cultural heritage , 2007 .

[60]  Francesco Visin,et al.  A guide to convolution arithmetic for deep learning , 2016, ArXiv.

[61]  C. Lerma,et al.  A discussion concerning active infrared thermography in the evaluation of buildings air infiltration , 2018, Energy and Buildings.

[62]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[63]  V. Paliouras,et al.  Simplified Hardware Implementation of the Softmax Activation Function , 2019, 2019 8th International Conference on Modern Circuits and Systems Technologies (MOCAST).

[64]  Solmoi Park,et al.  Defect identification in composite materials via thermography and deep learning techniques , 2020 .

[65]  Cao Vu Dung,et al.  Autonomous concrete crack detection using deep fully convolutional neural network , 2019, Automation in Construction.

[66]  Seoung Wug Oh,et al.  Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[67]  Glenn J Tattersall,et al.  Infrared thermography: A non-invasive window into thermal physiology. , 2016, Comparative biochemistry and physiology. Part A, Molecular & integrative physiology.

[68]  Chao Yang,et al.  A Survey on Deep Transfer Learning , 2018, ICANN.

[69]  이창기,et al.  Convolutional Neural Network를 이용한 한국어 영화평 감성 분석 , 2016 .

[70]  Ermanno G. Grinzato,et al.  Infrared thermographic nondestructive testing of frescos: Thermal modeling and image processing of three dimensional heat diffusion phenomena , 2002 .

[71]  Zhexiong Shang,et al.  Automatic delamination segmentation for bridge deck based on encoder-decoder deep learning through UAV-based thermography , 2020 .

[72]  Rik Van de Walle,et al.  Deep Learning for Infrared Thermal Image Based Machine Health Monitoring , 2017, IEEE/ASME Transactions on Mechatronics.

[73]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Afshin Marani,et al.  Mixture Optimization of Recycled Aggregate Concrete Using Hybrid Machine Learning Model , 2020, Materials.

[75]  Luca Maria Gambardella,et al.  Fast image scanning with deep max-pooling convolutional neural networks , 2013, 2013 IEEE International Conference on Image Processing.

[76]  Janet Blake,et al.  On Defining the Cultural Heritage , 2000, International and Comparative Law Quarterly.

[77]  Xavier Maldague,et al.  Qualitative Assessments via Infrared Vision of Sub-surface Defects Present Beneath Decorative Surface Coatings , 2017 .

[78]  Giorgio Baldinelli,et al.  A model for the improvement of thermal bridges quantitative assessment by infrared thermography , 2018 .

[79]  Xavier Maldague,et al.  Theory and Practice of Infrared Technology for Nondestructive Testing , 2001 .

[80]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[81]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[82]  Abhishek Dutta,et al.  The VIA Annotation Software for Images, Audio and Video , 2019, ACM Multimedia.

[83]  Richard J. Povinelli,et al.  Short-Term Load Forecasting of Natural Gas with Deep Neural Network Regression † , 2018, Energies.

[84]  Susana Lagüela-Lopez,et al.  Assessing Rebar Corrosion through the Combination of Nondestructive GPR and IRT Methodologies , 2019, Remote. Sens..

[85]  C. Ibarra-Castanedo,et al.  More than Fifty Shades of Grey: Quantitative Characterization of Defects and Interpretation Using SNR and CNR , 2018, Journal of Nondestructive Evaluation.

[86]  Murat Ceylan,et al.  Comparison of Traditional Transformations for Data Augmentation in Deep Learning of Medical Thermography , 2019, 2019 42nd International Conference on Telecommunications and Signal Processing (TSP).

[87]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[88]  Ali Kashif Bashir,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2013, ICIRA 2013.