ARCHITECTURAL HERITAGE RECOGNITION IN HISTORICAL FILM FOOTAGE USING NEURAL NETWORKS

Abstract. Researching historical archives for material suitable for photogrammetry is essential for the documentation and 3D reconstruction of Cultural Heritage, especially when this heritage has been lost or transformed over time. This research presents an innovative workflow which combines the photogrammetric procedure with Machine Learning for the processing of historical film footage. A Neural Network is trained to automatically detect frames in which architectural heritage appears. These frames are subsequently processed using photogrammetry and finally the resulting model is assessed for metric quality. This paper proposes best practises in training and validation on a Cultural Heritage asset. The algorithm was tested through a case study of the Tour Saint Jacques in Paris for which an entirely new dataset was created. The findings are encouraging both in terms of saving human effort and of improvement of the photogrammetric survey pipeline. This new tool can help researchers to better manage and organize historical information.

[1]  Jan-Michael Frahm,et al.  Structure-from-Motion Revisited , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  Jiri Matas,et al.  P-N learning: Bootstrapping binary classifiers by structural constraints , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  Roberto Medina,et al.  Classification of Architectural Heritage Images Using Deep Learning Techniques , 2017 .

[5]  Ankush Mittal,et al.  Image based Indian monument recognition using convoluted neural networks , 2017, 2017 International Conference on Big Data, IoT and Data Science (BID).

[6]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Hang Li,et al.  Convolutional Neural Network Architectures for Matching Natural Language Sentences , 2014, NIPS.

[8]  Ondrej Chum,et al.  CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples , 2016, ECCV.

[9]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[10]  Giuseppe Fiameni,et al.  A Performance Study of Machine and Deep Learning Frameworks on Cineca HPC Systems , 2017, PARCO.

[11]  Abdelhak Belhi,et al.  Leveraging Known Data for Missing Label Prediction in Cultural Heritage Context , 2018, Applied Sciences.

[12]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[13]  Abhishek Dutta,et al.  The VGG Image Annotator (VIA) , 2019, ArXiv.

[14]  Mattia D'Antonio,et al.  I-media-cities, a searchable platform on moving images with automatic and manual annotations , 2017, 2017 23rd International Conference on Virtual System & Multimedia (VSMM).

[15]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  V. Palma,et al.  TOWARDS DEEP LEARNING FOR ARCHITECTURE: A MONUMENT RECOGNITION MOBILE APP , 2019, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[17]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[19]  内山 庄一郎,et al.  SfM-MVS (Structure from Motion and multi-view stereo) 技術の地形計測への活用 , 2014 .

[20]  Fulvio Rinaudo,et al.  BENCHMARK OF METRIC QUALITY ASSESSMENT IN PHOTOGRAMMETRIC RECONSTRUCTION FOR HISTORICAL FILM FOOTAGE , 2019, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[21]  Ronan Sicre,et al.  Particular object retrieval with integral max-pooling of CNN activations , 2015, ICLR.