On the relation between landscape beauty and land cover: A case study in the U.K. at Sentinel-2 resolution with interpretable AI

Abstract The environment where we live and recreate can have a significant effect on our well-being. More beautiful landscapes have considerable benefits to both health and quality of life. When we chose where to live or our next holiday destination, we do so according to some perception of the environment around us. In a way, we value nature and assign an ecosystem service to it. Landscape aesthetics, or scenicness, is one such service, which we consider in this paper as a collective perceived quality. We present a deep learning model called ScenicNet for the large-scale inventorisation of landscape scenicness from satellite imagery. We model scenicness with an interpretable deep learning model and learn a landscape beauty estimator based on crowdsourced scores derived from more than two hundred thousand landscape images in the United Kingdom. Our ScenicNet model learns the relationship between land cover types and scenicness by using land cover prediction as an interpretable intermediate task to scenicness regression. It predicts landscape scenicness and land cover from the Corine Land Cover product concurrently, without compromising the accuracy of either task. In addition, our proposed model is interpretable in the sense that it learns to express preferences for certain types of land covers in a manner that is easily understandable by an end-user. Our semantic bottleneck also allows us to further our understanding of crowd preferences for landscape aesthetics.

[1]  Nicolas Courty,et al.  Contextual Semantic Interpretability , 2020, ACCV.

[2]  Gary Fry,et al.  Health effects of viewing landscapes - Landscape types in environmental psychology , 2007 .

[3]  Alexander Binder,et al.  Unmasking Clever Hans predictors and assessing what machines really learn , 2019, Nature Communications.

[4]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  T. Daniel,et al.  Foundations for an Ecological Aesthetic: Can Information Alter Landscape Preferences? , 2007 .

[6]  Michele Volpi,et al.  Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Pierre Alliez,et al.  High-Resolution Aerial Image Labeling With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[8]  LIII , 2018, Out of the Shadow.

[9]  Klaus-Robert Müller,et al.  Towards Explainable Artificial Intelligence , 2019, Explainable AI.

[10]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[11]  Xiao Xiang Zhu,et al.  Deep learning in remote sensing: a review , 2017, ArXiv.

[12]  Jing Huang,et al.  DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[13]  Michele Volpi,et al.  Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[14]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[15]  Erwan Scornet,et al.  A random forest guided tour , 2015, TEST.

[16]  Alexandre Boulch,et al.  Processing of Extremely High-Resolution LiDAR and RGB Data: Outcome of the 2015 IEEE GRSS Data Fusion Contest–Part A: 2-D Contest , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[17]  D. Moran,et al.  What does the public want from agriculture and the countryside? A review of evidence and methods , 2004 .

[18]  Robert L. Thayer,et al.  Implied human influence reduces landscape beauty , 1980 .

[19]  Nicolas Audebert,et al.  Deep Learning for Classification of Hyperspectral Data: A Comparative Review , 2019, IEEE Geoscience and Remote Sensing Magazine.

[20]  J. Palmer.  Using spatial metrics to predict scenic perception in a changing landscape: Dennis, Massachusetts , 2004 .

[21]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[22]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[23]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[24]  Devis Tuia,et al.  Defining and spatially modelling cultural ecosystem services using crowdsourced data , 2020 .

[25]  Uwe Stilla,et al.  Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection , 2016, ISPRS Journal of Photogrammetry and Remote Sensing.

[26]  E. Anthony,et al.  Disentangling the relative impacts of climate change and human activities on fluvial sediment supply to the coast by the world’s large rivers: Pearl River Basin, China , 2019, Scientific Reports.

[27]  Alexander Dekhtyar,et al.  Information Retrieval , 2018, Lecture Notes in Computer Science.

[28]  Bart Baesens,et al.  An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models , 2011, Decis. Support Syst..

[29]  Tim Miller,et al.  Explanation in Artificial Intelligence: Insights from the Social Sciences , 2017, Artif. Intell..

[30]  T. Daniel,et al.  Measuring landscape esthetics: the scenic beauty estimation method , 1976 .

[31]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[32]  Rich Caruana,et al.  Multitask Learning , 1997, Machine Learning.

[33]  Faiz Ur Rahman,et al.  Aerial-CAM: Salient Structures and Textures in Network Class Activation Maps of Aerial Imagery , 2018, 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP).

[34]  Tobias Preis,et al.  Happiness is Greater in More Scenic Locations , 2019, Scientific Reports.

[35]  Devis Tuia,et al.  Semantically Interpretable Activation Maps: what-where-how explanations within CNNs , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[36]  Wei Dai,et al.  MultiCAM: Multiple Class Activation Mapping for Aircraft Recognition in Remote Sensing Images , 2019, Remote. Sens..

[37]  Grete Grindal Patil,et al.  Biophilia: Does Visual Contact with Nature Impact on Health and Well-Being? , 2009, International journal of environmental research and public health.

[38]  Scott Workman,et al.  Understanding and Mapping Natural Beauty , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  D. Tuia,et al.  Interpretable Scenicness from Sentinel-2 Imagery , 2020, IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium.

[40]  R. Costanza,et al.  Contributions of cultural services to the ecosystem services agenda , 2012, Proceedings of the National Academy of Sciences.

[41]  Mirjam de Groot,et al.  Eyesores in sight: Quantifying the impact of man-made elements on the scenic beauty of Dutch landscapes , 2012 .

[42]  Ryosuke Nakamura,et al.  Solar Power Plant Detection on Multi-Spectral Satellite Imagery using Weakly-Supervised CNN with Feedback Features and m-PCNN Fusion , 2017, BMVC.

[43]  Begüm Demir,et al.  Bigearthnet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[44]  Petra Lindemann-Matthies,et al.  Aesthetic preference for a Swiss alpine landscape: the impact of different agricultural land-use with different biodiversity. , 2010 .

[45]  Terry C. Daniel,et al.  Progress in Predicting the Perceived Scenic Beauty of Forest Landscapes , 1981 .

[46]  Rafael Molina,et al.  Deep Gaussian processes for biogeophysical parameter retrieval and model inversion , 2020, ISPRS journal of photogrammetry and remote sensing : official publication of the International Society for Photogrammetry and Remote Sensing.

[47]  Paulo Pereira,et al.  Corrigendum: Gap junctional protein Cx43 is involved in the communication between extracellular vesicles and mammalian cells , 2015, Scientific Reports.

[48]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Gustavo Camps-Valls,et al.  Learning Relevant Image Features With Multiple-Kernel Classification , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[50]  Erich Tasser,et al.  Predicting scenic beauty of mountain regions , 2013 .

[51]  J. Krippendorf.  Die Ferienmenschen : für ein neues Verständnis von Freizeit und Reisen , 1986 .

[52]  Tobias Preis,et al.  Using deep learning to quantify the beauty of outdoor places , 2017, Royal Society Open Science.

[53]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[54]  Tobias Preis,et al.  Quantifying the Impact of Scenic Environments on Health , 2015, Scientific Reports.