An Unsupervised Remote Sensing Single-Image Super-Resolution Method Based on Generative Adversarial Network

Image super-resolution (SR) technique can improve the spatial resolution of images without upgrading the imaging system. As a result, SR promotes the development of high resolution (HR) remote sensing image applications. Many remote sensing image SR algorithms based on deep learning have been proposed recently, which can effectively improve the spatial resolution under the constraints of HR images. However, images acquired by remote sensing imaging devices typically have lower resolution. Hence, an insufficient number of HR remote sensing images are available for training deep neural networks. In view of this problem, we propose an unsupervised SR method that does not require HR remote sensing images. The proposed method introduces a generative adversarial network (GAN) that obtains SR images through the generator; then, the SR images are downsampled to train the discriminator with low resolution (LR) images. Our method outperformed several methods in terms of the quality of the obtained SR images as measured by 6 evaluation metrics, which proves the satisfactory performance of the proposed unsupervised method for improving the spatial resolution of remote sensing images.

[1]  Zongxu Pan,et al.  Super-Resolution of Remote Sensing Images Based on Transferred Generative Adversarial Network , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[2]  Qinghua Hu,et al.  Unsupervised Degradation Learning for Single Image Super-Resolution , 2018, ArXiv.

[3]  Eric Dubois,et al.  Image up-sampling using total-variation regularization with a new observation model , 2005, IEEE Transactions on Image Processing.

[4]  Qian Du,et al.  Hyperspectral image super-resolution via convolutional neural network , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[5]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Ling Shao,et al.  Single satellite imagery simultaneous super-resolution and colorization using multi-task deep neural networks , 2018, J. Vis. Commun. Image Represent..

[7]  Luc Van Gool,et al.  Jointly Optimized Regressors for Image Super‐resolution , 2015, Comput. Graph. Forum.

[8]  J. L. Harris,et al.  Diffraction and Resolving Power , 1964 .

[9]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[10]  Biao Hou,et al.  Structural-Correlated Self-Examples Based Superresolution of Single Remote Sensing Image , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[11]  Seungyong Lee,et al.  SRFeat: Single Image Super-Resolution with Feature Discrimination , 2018, ECCV.

[12]  Zhongyuan Wang,et al.  Video Satellite Imagery Super Resolution via Convolutional Neural Networks , 2017, IEEE Geoscience and Remote Sensing Letters.

[13]  Kyung-Ah Sohn,et al.  Fast, Accurate, and, Lightweight Super-Resolution with Cascading Residual Network , 2018, ECCV.

[14]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Jie Li,et al.  Image super-resolution: The techniques, applications, and future , 2016, Signal Process..

[16]  Xinbo Gao,et al.  Fast and Accurate Single Image Super-Resolution via Information Distillation Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Siyuan Liu,et al.  Unsupervised Image Super-Resolution Using Cycle-in-Cycle Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[18]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[19]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[21]  Aurélien Ducournau,et al.  Deep learning for ocean remote sensing: an application of convolutional neural networks for super-resolution on satellite-derived SST data , 2016, 2016 9th IAPR Workshop on Pattern Recogniton in Remote Sensing (PRRS).

[22]  Zhenwei Shi,et al.  Super-Resolution for Remote Sensing Images via Local–Global Combined Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[23]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[24]  Michal Irani,et al.  Blind Deblurring Using Internal Patch Recurrence , 2014, ECCV.

[25]  Bhabatosh Chanda,et al.  Image Upscaling Using Multiple Dictionaries of Natural Image Patches , 2012, ACCV.

[26]  Hua Huang,et al.  Neighbor embedding based super-resolution algorithm through edge detection and feature selection , 2009, Pattern Recognit. Lett..

[27]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[28]  Zongxu Pan,et al.  Achieving Super-Resolution Remote Sensing Images via the Wavelet Transform Combined With the Recursive Res-Net , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[29]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[30]  Wei Cui,et al.  Application of a Hybrid Model Based on a Convolutional Auto-Encoder and Convolutional Neural Network in Object-Oriented Remote Sensing Classification , 2018, Algorithms.

[31]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[32]  M. Wilscy,et al.  Single image super-resolution based on compressive sensing and TV minimization sparse recovery for remote sensing images , 2013, 2013 IEEE Recent Advances in Intelligent Computational Systems (RAICS).

[33]  Jean-Yves Tourneret,et al.  Single image super-resolution of medical ultrasound images using a fast algorithm , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[34]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[35]  Zongxu Pan,et al.  Super-Resolution Based on Compressive Sensing and Structural Self-Similarity for Remote Sensing Images , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Lorenzo Bruzzone,et al.  A Deep Network Architecture for Super-Resolution-Aided Hyperspectral Image Classification With Classwise Loss , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Narendra Ahuja,et al.  Single image super-resolution from transformed self-exemplars , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Lei Zhang,et al.  Image Deblurring and Super-Resolution by Adaptive Sparse Domain Selection and Adaptive Regularization , 2010, IEEE Transactions on Image Processing.

[40]  Filiberto Pla,et al.  A New Deep Generative Network for Unsupervised Remote Sensing Single-Image Super-Resolution , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[41]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[42]  Ken Turkowski,et al.  Filters for common resampling tasks , 1990 .

[43]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[44]  Michal Irani,et al.  "Zero-Shot" Super-Resolution Using Deep Internal Learning , 2017, CVPR.

[45]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Jian Yang,et al.  MemNet: A Persistent Memory Network for Image Restoration , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[47]  Peyman Milanfar,et al.  A wavelet-based interpolation-restoration method for superresolution (wavelet superresolution) , 2000 .

[48]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[49]  Chi-Keung Tang,et al.  Fast image/video upsampling , 2008, SIGGRAPH Asia '08.

[50]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Khizar Hayat,et al.  Super-Resolution via Deep Learning , 2017, Digit. Signal Process..

[52]  Fatos T. Yarman-Vural,et al.  Representation learning with convolutional sparse autoencoders for remote sensing , 2013, 2013 21st Signal Processing and Communications Applications Conference (SIU).

[53]  Shawn D. Newsam,et al.  Bag-of-visual-words and spatial extensions for land-use classification , 2010, GIS '10.

[54]  Zongxu Pan,et al.  Super-Resolution of Single Remote Sensing Image Based on Residual Dense Backprojection Networks , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[55]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[57]  Seunghyeon Rhee,et al.  Discrete cosine transform based regularized high-resolution image reconstruction algorithm , 1999 .

[58]  Biao Hou,et al.  Adaptive Super-Resolution for Remote Sensing Images Based on Sparse Representation With Global Joint Dictionary Model , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[59]  Nenghai Yu,et al.  StyleBank: An Explicit Representation for Neural Image Style Transfer , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Andrea Garzelli,et al.  Hypercomplex Quality Assessment of Multi/Hyperspectral Images , 2009, IEEE Geoscience and Remote Sensing Letters.

[61]  Yunsong Li,et al.  A spatial constraint and deep learning based hyperspectral image super-resolution method , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[62]  Xiaoqiang Lu,et al.  Remote Sensing Image Scene Classification: Benchmark and State of the Art , 2017, Proceedings of the IEEE.

[63]  Daniel Rueckert,et al.  Cardiac Image Super-Resolution with Global Correspondence Using Multi-Atlas PatchMatch , 2013, MICCAI.

[64]  Jing Yang,et al.  To learn image super-resolution, use a GAN to learn how to do image degradation first , 2018, ECCV.

[65]  Kangfu Mei,et al.  Multi-scale Residual Network for Image Super-Resolution , 2018, ECCV.

[66]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.

[67]  Michal Irani,et al.  Space-time super-resolution from a single video , 2011, CVPR 2011.

[68]  J. Goodman.  Introduction to Fourier optics , 1969 .

[69]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[70]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Wan-Chi Siu,et al.  Single image super-resolution using Gaussian process regression , 2011, CVPR 2011.

[72]  Filiberto Pla,et al.  Single-frame super-resolution in remote sensing: a practical overview , 2017 .

[73]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Yu-Bin Yang,et al.  Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections , 2016, ArXiv.

[76]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[77]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[78]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[79]  Gregory Shakhnarovich,et al.  Deep Back-Projection Networks for Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[80]  Dengxin Dai,et al.  Satellite Image Classification via Two-Layer Sparse Coding With Biased Image Representation , 2011, IEEE Geoscience and Remote Sensing Letters.

[81]  L. Lucy.  An iterative technique for the rectification of observed distributions , 1974 .

[82]  Mihai Datcu,et al.  A convolutional deep belief network for polarimetric SAR data feature extraction , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[83]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.