AFN: Attentional Feedback Network based 3D Terrain Super-Resolution

Terrain, representing features of an earth surface, plays a crucial role in many applications such as simulations, route planning, analysis of surface dynamics, computer graphics-based games, entertainment, films, to name a few. With recent advancements in digital technology, these applications demand the presence of high-resolution details in the terrain. In this paper, we propose a novel fully convolutional neural network-based super-resolution architecture to increase the resolution of low-resolution Digital Elevation Model (LRDEM) with the help of information extracted from the corresponding aerial image as a complementary modality. We perform the super-resolution of LRDEM using an attention-based feedback mechanism named 'Attentional Feedback Network' (AFN), which selectively fuses the information from LRDEM and aerial image to enhance and infuse the high-frequency features and to produce the terrain realistically. We compare the proposed architecture with existing state-of-the-art DEM super-resolution methods and show that the proposed architecture outperforms enhancing the resolution of input LRDEM accurately and in a realistic manner.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Bedrich Benes,et al.  Layered data representation for visual simulation of terrain erosion , 2001, Proceedings Spring Conference on Computer Graphics.

[3]  James M. Rehg,et al.  Terrain Synthesis from Digital Elevation Models , 2007, IEEE Transactions on Visualization and Computer Graphics.

[4]  Wolfgang Straßer,et al.  Terrain sketching , 2009, I3D '09.

[5]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[6]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Stephen Lin,et al.  Super resolution using edge prior and single image detail synthesis , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Marie-Paule Cani,et al.  A Review of Digital Terrain Modeling , 2019, Comput. Graph. Forum.

[9]  Marie-Paule Cani,et al.  Authoring landscapes by combining ecosystem and terrain erosion simulation , 2017, ACM Trans. Graph..

[10]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Kwang In Kim,et al.  Single-Image Super-Resolution Using Sparse Regression and Natural Image Prior , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Yuan Xie,et al.  Instance-Level Salient Object Segmentation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Thomas S. Huang,et al.  Deep Networks for Image Super-Resolution with Sparse Prior , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[14]  Bedrich Benes,et al.  Terrain Modelling from Feature Primitives , 2015, Comput. Graph. Forum.

[15]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[16]  Wei Wu,et al.  Feedback Network for Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[18]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Lin Sun,et al.  Feedback Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Ken Perlin,et al.  An image synthesizer , 1988 .

[21]  Ke Zhang,et al.  Residual Networks of Residual Networks: Multilevel Residual Networks , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[22]  Avinash Sharma,et al.  Feedback Neural Network based Super-resolution of DEM for generating high fidelity features , 2020, ArXiv.

[23]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[24]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[25]  Christian Wolf,et al.  Interactive example-based terrain authoring with conditional generative adversarial networks , 2017, ACM Trans. Graph..

[26]  Ping Wah Wong,et al.  Edge-directed interpolation , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[27]  Bedrich Benes,et al.  Terrain generation using procedural models based on hydrology , 2013, ACM Trans. Graph..

[28]  Carlos Andújar,et al.  Terrain Super‐resolution through Aerial Imagery and Fully Convolutional Networks , 2018, Comput. Graph. Forum.

[29]  Narendra Ahuja,et al.  Single image super-resolution from transformed self-exemplars , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  R. Westermann,et al.  Real-Time Editing , Synthesis , and Rendering of Infinite Landscapes on GPUs , 2006 .

[31]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Samir Akkouche,et al.  Feature based terrain generation using diffusion equation , 2010, Comput. Graph. Forum.

[33]  Lei Zhang,et al.  Convolutional Sparse Coding for Image Super-Resolution , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[34]  Eric Galin,et al.  Sparse representation of terrains for procedural modeling , 2016, Comput. Graph. Forum.

[35]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[36]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[37]  F. Kenton Musgrave,et al.  The synthesis and rendering of eroded fractal terrains , 1989, SIGGRAPH.