A Neural Networks Approach to Detecting Lost Heritage in Historical Video

Documenting Cultural Heritage through the extraction of 3D measures with photogrammetry is fundamental for the conservation of the memory of the past. However, when the heritage has been lost the only way to recover this information is the use of historical images from archives. The aim of this study is to experiment with new ways to search for architectural heritage in video material and to save the effort of the operator in the archive in terms of efficiency and time. A workflow is proposed to automatically detect lost heritage in film footage using Deep Learning to find suitable images to process with photogrammetry for its 3D virtual reconstruction. The performance of the network was tested on two case studies considering different architectural scenarios, the Tour Saint Jacques which still exists for the tuning of the networks, and Les Halles to test the algorithms on a real case of an architecture which has been destroyed. Despite the poor quantity and low quality of the historical images available for the training of the network, it has been demonstrated that, with few frames, it was possible to reach the same results in terms of performance of a network trained on a large dataset. Moreover, with the introduction of new metrics based on time intervals the measure of the real time saving in terms of human effort was achieved. These findings represent an important innovation in the documentation of destroyed monuments and open new ways to recover information about the past.

[1]  Abdelhak Belhi,et al.  Leveraging Known Data for Missing Label Prediction in Cultural Heritage Context , 2018, Applied Sciences.

[2]  Lauren M. O'Connell,et al.  Afterlives of the Tour Saint-Jacques: Plotting the Perceptual History of an Urban Fragment , 2001 .

[3]  David Picard,et al.  Challenges in Content-Based Image Indexing of Cultural Heritage Collections , 2015, IEEE Signal Processing Magazine.

[4]  Fulvio Rinaudo,et al.  BENCHMARK OF METRIC QUALITY ASSESSMENT IN PHOTOGRAMMETRIC RECONSTRUCTION FOR HISTORICAL FILM FOOTAGE , 2019, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[5]  Martyna Poreba,et al.  Challenging Deep Image Descriptors for Retrieval in Heterogeneous Iconographic Collections , 2019, SUMAC @ ACM Multimedia.

[6]  Chris Bowerman,et al.  Saving Cultural Heritage with Digital Make-Believe: Machine Learning and Digital Techniques to the Rescue , 2017, BCS HCI.

[7]  Ondrej Chum,et al.  CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples , 2016, ECCV.

[8]  Abhishek Dutta,et al.  The VIA Annotation Software for Images, Audio and Video , 2019, ACM Multimedia.

[9]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[10]  Jaap Kamps,et al.  Deep Learning as a Tool for Early Cinema Analysis , 2019, SUMAC @ ACM Multimedia.

[11]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Roberto Medina,et al.  Classification of Architectural Heritage Images Using Deep Learning Techniques , 2017 .

[13]  Mattia D'Antonio,et al.  I-media-cities, a searchable platform on moving images with automatic and manual annotations , 2017, 2017 23rd International Conference on Virtual System & Multimedia (VSMM).

[14]  Jenny Benois-Pineau,et al.  Organizing Cultural Heritage with Deep Features , 2019, SUMAC @ ACM Multimedia.

[15]  Ankush Mittal,et al.  Image based Indian monument recognition using convoluted neural networks , 2017, 2017 International Conference on Big Data, IoT and Data Science (BID).

[16]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[18]  Fulvio Rinaudo,et al.  ARCHITECTURAL HERITAGE RECOGNITION IN HISTORICAL FILM FOOTAGE USING NEURAL NETWORKS , 2019, ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.