Edge-Enhanced GAN for Remote Sensing Image Superresolution

The current superresolution (SR) methods based on deep learning have shown remarkable comparative advantages but remain unsatisfactory in recovering the high-frequency edge details of the images in noise-contaminated imaging conditions, e.g., remote sensing satellite imaging. In this paper, we propose a generative adversarial network (GAN)-based edge-enhancement network (EEGAN) for robust satellite image SR reconstruction along with the adversarial learning strategy that is insensitive to noise. In particular, EEGAN consists of two main subnetworks: an ultradense subnetwork (UDSN) and an edge-enhancement subnetwork (EESN). In UDSN, a group of 2-D dense blocks is assembled for feature extraction and to obtain an intermediate high-resolution result that looks sharp but is eroded with artifacts and noises as previous GAN-based methods do. Then, EESN is constructed to extract and enhance the image contours by purifying the noise-contaminated components with mask processing. The recovered intermediate image and enhanced edges can be combined to generate the result that enjoys high credibility and clear contents. Extensive experiments on Kaggle Open Source Data set, Jilin-1 video satellite images, and Digitalglobe show superior reconstruction performance compared to the state-of-the-art SR approaches.

[1]  Renjie Liao,et al.  Deep Edge-Aware Filters , 2015, ICML.

[2]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  Xianming Liu,et al.  Hyperspectral Image Classification in the Presence of Noisy Labels , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Shuyuan Yang,et al.  Novel Super Resolution Restoration of Remote Sensing Images Based on Compressive Sensing and Example Patches-Aided Dictionary Learning , 2011, 2011 International Workshop on Multi-Platform/Multi-Sensor Remote Sensing and Mapping.

[5]  Peng Li,et al.  SegGAN: Semantic Segmentation with Generative Adversarial Network , 2018, 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM).

[6]  Huchuan Lu,et al.  Learning Dual Convolutional Neural Networks for Low-Level Vision , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Tao Lu,et al.  Multi-Memory Convolutional Neural Network for Video Super-Resolution , 2019, IEEE Transactions on Image Processing.

[8]  Carlos López-Martínez,et al.  Edge Enhancement Algorithm Based on the Wavelet Transform for Automatic Edge Detection in SAR Images , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[9]  Kilian Q. Weinberger,et al.  Multi-Scale Dense Networks for Resource Efficient Image Classification , 2017, ICLR.

[10]  Shuiping Gou,et al.  Remote Sensing Image Super-Resolution Reconstruction Based on Nonlocal Pairwise Dictionaries and Double Regularization , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[11]  Jiayi Ma,et al.  Hyperspectral Unmixing with Robust Collaborative Sparse Regression , 2016, Remote. Sens..

[12]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[14]  Yuan Yao,et al.  Deep Distillation Recursive Network for Remote Sensing Imagery Super-Resolution , 2018, Remote. Sens..

[15]  Shutao Li,et al.  Remote Sensing Scene Classification Using Multilayer Stacked Covariance Pooling , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[17]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Jiashi Feng,et al.  Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution. , 2017, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[19]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Karen Anderson,et al.  Tracking Fine-Scale Structural Changes in Coastal Dune Morphology Using Kite Aerial Photography and Uncertainty-Assessed Structure-from-Motion Photogrammetry , 2018, Remote. Sens..

[22]  Shutao Li,et al.  Hyperspectral Image Classification With Squeeze Multibias Network , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[23]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Guangyu Sun,et al.  SRPGAN: Perceptual Generative Adversarial Network for Single Image Super Resolution , 2017, ArXiv.

[27]  Zhongyuan Wang,et al.  Video Satellite Imagery Super Resolution via Convolutional Neural Networks , 2017, IEEE Geoscience and Remote Sensing Letters.

[28]  Junjun Jiang,et al.  Multiscale Union Regions Adaptive Sparse Representation for Hyperspectral Image Classification , 2017, Remote. Sens..

[29]  Bernhard Schölkopf,et al.  EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Jian Zhu,et al.  Deformable Convolutional Neural Networks for Hyperspectral Image Classification , 2018, IEEE Geoscience and Remote Sensing Letters.

[31]  Azriel Rosenfeld,et al.  Optimally isotropic Laplacian operator , 1999, IEEE Trans. Image Process..

[32]  Yi Yu,et al.  Spatial-Aware Collaborative Representation for Hyperspectral Remote Sensing Image Classification , 2017, IEEE Geoscience and Remote Sensing Letters.

[33]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Jürgen Schmidhuber,et al.  Training Very Deep Networks , 2015, NIPS.

[35]  Feng Li,et al.  Universal HMT based super resolution for remote sensing images , 2008, 2008 15th IEEE International Conference on Image Processing.

[36]  Rui Bian,et al.  A Novel Stochastic Stratified Average Gradient Method: Convergence Rate and Its Complexity , 2017, 2018 International Joint Conference on Neural Networks (IJCNN).

[37]  Volodymyr Ponomaryov,et al.  Super Resolution Image Generation Using Wavelet Domain Interpolation With Edge Extraction via a Sparse Representation , 2014, IEEE Geoscience and Remote Sensing Letters.

[38]  Zixiang Xiong,et al.  Separability and Compactness Network for Image Recognition and Superresolution , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[39]  Cheng Xiansheng,et al.  An edge detection new algorithm based on Laplacian Operator , 2011, 2011 IEEE 3rd International Conference on Communication Software and Networks.

[40]  Ming-Hsuan Yang,et al.  Learning Recursive Filters for Low-Level Vision via a Hybrid Neural Network , 2016, ECCV.

[41]  Wen Gao,et al.  Depth Super-Resolution via Joint Color-Guided Internal and External Regularizations , 2019, IEEE Transactions on Image Processing.

[42]  Lei Zhang,et al.  Canny edge detection enhancement by scale multiplication , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[44]  Jun Li,et al.  Unsupervised Feature Extraction in Hyperspectral Images Based on Wasserstein Generative Adversarial Network , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[45]  Guangming Shi,et al.  Hyperspectral Image Super-Resolution via Non-Negative Structured Sparse Representation. , 2016, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[46]  Thomas R. Walter,et al.  Radar Path Delay Effects in Volcanic Gas Plumes: The Case of Láscar Volcano, Northern Chile , 2018, Remote. Sens..

[47]  Lizhe Wang,et al.  SuperPCA: A Superpixelwise PCA Approach for Unsupervised Feature Extraction of Hyperspectral Imagery , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[48]  Jian Yang,et al.  MemNet: A Persistent Memory Network for Image Restoration , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[49]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[51]  Junjun Jiang,et al.  A Progressively Enhanced Network for Video Satellite Imagery Superresolution , 2018, IEEE Signal Processing Letters.

[52]  Wen Gao,et al.  Depth Restoration From RGB-D Data via Joint Adaptive Regularization and Thresholding on Manifolds , 2019, IEEE Transactions on Image Processing.

[53]  Lucas Theis,et al.  Amortised MAP Inference for Image Super-resolution , 2016, ICLR.

[54]  Jorge Núñez,et al.  Super-Resolution of Remotely Sensed Images With Variable-Pixel Linear Reconstruction , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[55]  Xin Wang,et al.  Laplacian Operator-Based Edge Detectors , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Zhenfeng Shao,et al.  Remote Sensing Image Fusion With Deep Convolutional Neural Network , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[57]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[58]  Xinfeng Zhang,et al.  Enhanced Image Decoding via Edge-Preserving Generative Adversarial Networks , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[59]  Zhenwei Shi,et al.  Super-Resolution for Remote Sensing Images via Local–Global Combined Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[60]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.