Satellite Image Super-Resolution via Multi-Scale Residual Deep Neural Network

Recently, the application of satellite remote sensing images is becoming increasingly popular, but the observed images from satellite sensors are frequently in low-resolution (LR). Thus, they cannot fully meet the requirements of object identification and analysis. To utilize the multi-scale characteristics of objects fully in remote sensing images, this paper presents a multi-scale residual neural network (MRNN). MRNN adopts the multi-scale nature of satellite images to reconstruct high-frequency information accurately for super-resolution (SR) satellite imagery. Different sizes of patches from LR satellite images are initially extracted to fit different scale of objects. Large-, middle-, and small-scale deep residual neural networks are designed to simulate differently sized receptive fields for acquiring relative global, contextual, and local information for prior representation. Then, a fusion network is used to refine different scales of information. MRNN fuses the complementary high-frequency information from differently scaled networks to reconstruct the desired high-resolution satellite object image, which is in line with human visual experience (“look in multi-scale to see better”). Experimental results on the SpaceNet satellite image and NWPU-RESISC45 databases show that the proposed approach outperformed several state-of-the-art SR algorithms in terms of objective and subjective image qualities.

[1]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[2]  Andrea Garzelli,et al.  A Review of Image Fusion Algorithms Based on the Super-Resolution Paradigm , 2016, Remote. Sens..

[3]  Jiaying Liu,et al.  Exploiting multi-scale spatial structures for sparsity based single image super-resolution , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[4]  Roger Y. Tsai,et al.  Multiframe image restoration and registration , 1984 .

[5]  Hua Huang,et al.  Super-Resolution Method for Face Recognition Using Nonlinear Mappings on Coherent Features , 2011, IEEE Transactions on Neural Networks.

[6]  Shuang Wang,et al.  How Does the Low-Rank Matrix Decomposition Help Internal and External Learnings for Super-Resolution , 2016, IEEE Transactions on Image Processing.

[7]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Tom Hartley,et al.  Low-level properties of natural images predict topographic patterns of neural response in the ventral visual pathway. , 2015, Journal of vision.

[9]  Chinh T. Dang,et al.  Fast Single-Image Super-Resolution Via Tangent Space Learning of High-Resolution-Patch Manifold , 2017, IEEE Transactions on Computational Imaging.

[10]  Tingting Lv,et al.  Detecting Building Edges from High Spatial Resolution Remote Sensing Imagery Using Richer Convolution Features Network , 2018, Remote. Sens..

[11]  Junjun Jiang,et al.  Locality Preserving Matching , 2018, International Journal of Computer Vision.

[12]  Zhenfeng Shao,et al.  Remote Sensing Image Fusion With Deep Convolutional Neural Network , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[13]  Stephan Tschechne,et al.  Hierarchical representation of shapes in visual cortex—from localized features to figural shape segregation , 2014, Front. Comput. Neurosci..

[14]  Aggelos K. Katsaggelos,et al.  Video Super-Resolution With Convolutional Neural Networks , 2016, IEEE Transactions on Computational Imaging.

[15]  S. Süsstrunk,et al.  Frequency-tuned salient region detection , 2009, CVPR 2009.

[16]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[17]  Wenhan Yang,et al.  Dmcnn: Dual-Domain Multi-Scale Convolutional Neural Network for Compression Artifacts Removal , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[18]  Liangpei Zhang,et al.  Remote Sensing Image Subpixel Mapping Based on Adaptive Differential Evolution , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[19]  Yifan He,et al.  Single Image Super-Resolution Based on Multi-Scale Competitive Convolutional Neural Network , 2018, Sensors.

[20]  Junjun Jiang,et al.  Edge-Enhanced GAN for Remote Sensing Image Superresolution , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[21]  Xiaoqiang Lu,et al.  Remote Sensing Image Scene Classification: Benchmark and State of the Art , 2017, Proceedings of the IEEE.

[22]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[23]  Thomas B. Moeslund,et al.  Super-resolution: a comprehensive survey , 2014, Machine Vision and Applications.

[24]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[25]  Kiyoharu Aizawa,et al.  Category-Based Deep CCA for Fine-Grained Venue Discovery From Multimodal Data , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Tao Lu,et al.  Face super-resolution via bilayer contextual representation , 2019, Signal Process. Image Commun..

[27]  Xuelong Li,et al.  Single Image Super-Resolution With Multiscale Similarity Learning , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Jinhui Tang,et al.  Richer Convolutional Features for Edge Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Andreas K. Maier,et al.  Robust Multiframe Super-Resolution Employing Iteratively Re-Weighted Minimization , 2016, IEEE Transactions on Computational Imaging.

[31]  Bo Huang,et al.  Support Vector Regression-Based Downscaling for Intercalibration of Multiresolution Satellite Images , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[32]  Narendra Ahuja,et al.  Single image super-resolution from transformed self-exemplars , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Mohammad Alam,et al.  Superresolution Construction of Multispectral Imagery Based on Local Enhancement , 2008, IEEE Geoscience and Remote Sensing Letters.

[34]  Lei Chen,et al.  Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval , 2017, ACM Trans. Multim. Comput. Commun. Appl..

[35]  Liangpei Zhang,et al.  A MAP Approach for Joint Motion Estimation, Segmentation, and Super Resolution , 2007, IEEE Transactions on Image Processing.

[36]  Weixing Zhang,et al.  Deep Convolutional Neural Networks for Automated Characterization of Arctic Ice-Wedge Polygons in Very High Spatial Resolution Aerial Imagery , 2018, Remote. Sens..

[37]  Shuai Li,et al.  End-to-End Airport Detection in Remote Sensing Images Combining Cascade Region Proposal Networks and Multi-Threshold Detection Networks , 2018, Remote. Sens..

[38]  Tao Mei,et al.  Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[40]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Junjun Jiang,et al.  Rectangular-Normalized Superpixel Entropy Index for Image Quality Assessment , 2018, Entropy.

[43]  Tao Lu,et al.  Multi-Memory Convolutional Neural Network for Video Super-Resolution , 2019, IEEE Transactions on Image Processing.

[44]  Kai Liu,et al.  Multi-Stream Convolutional Neural Network for SAR Automatic Target Recognition , 2018, Remote. Sens..

[45]  Zhongyuan Wang,et al.  Video Satellite Imagery Super Resolution via Convolutional Neural Networks , 2017, IEEE Geoscience and Remote Sensing Letters.

[46]  Jie Li,et al.  Image super-resolution: The techniques, applications, and future , 2016, Signal Process..

[47]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[48]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Peyman Milanfar,et al.  RAISR: Rapid and Accurate Image Super Resolution , 2016, IEEE Transactions on Computational Imaging.

[51]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).