Deep Distillation Recursive Network for Remote Sensing Imagery Super-Resolution

Deep convolutional neural networks (CNNs) have been widely used and achieved state-of-the-art performance in many image or video processing and analysis tasks. In particular, for image super-resolution (SR) processing, previous CNN-based methods have led to significant improvements, when compared with shallow learning-based methods. However, previous CNN-based algorithms with simple direct or skip connections are of poor performance when applied to remote sensing satellite images SR. In this study, a simple but effective CNN framework, namely deep distillation recursive network (DDRN), is presented for video satellite image SR. DDRN includes a group of ultra-dense residual blocks (UDB), a multi-scale purification unit (MSPU), and a reconstruction module. In particular, through the addition of rich interactive links in and between multiple-path units in each UDB, features extracted from multiple parallel convolution layers can be shared effectively. Compared with classical dense-connection-based models, DDRN possesses the following main properties. (1) DDRN contains more linking nodes with the same convolution layers. (2) A distillation and compensation mechanism, which performs feature distillation and compensation in different stages of the network, is also constructed. In particular, the high-frequency components lost during information propagation can be compensated in MSPU. (3) The final SR image can benefit from the feature maps extracted from UDB and the compensated components obtained from MSPU. Experiments on Kaggle Open Source Dataset and Jilin-1 video satellite images illustrate that DDRN outperforms the conventional CNN-based baselines and some state-of-the-art feature extraction approaches.

[1]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[3]  Rui Bian,et al.  A Novel Stochastic Stratified Average Gradient Method: Convergence Rate and Its Complexity , 2017, 2018 International Joint Conference on Neural Networks (IJCNN).

[4]  Lei Zhang,et al.  Convolutional Sparse Coding for Image Super-Resolution , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Thomas S. Huang,et al.  Image super-resolution as sparse representation of raw image patches , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Kilian Q. Weinberger,et al.  Multi-Scale Dense Networks for Resource Efficient Image Classification , 2017, ICLR.

[8]  Guangming Shi,et al.  Hyperspectral Image Super-Resolution via Non-Negative Structured Sparse Representation. , 2016, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[9]  Narendra Ahuja,et al.  Single image super-resolution from transformed self-exemplars , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[11]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Kai Liu,et al.  Multi-Stream Convolutional Neural Network for SAR Automatic Target Recognition , 2018, Remote. Sens..

[13]  Jiashi Feng,et al.  Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution. , 2017, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[14]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Chih-Yuan Yang,et al.  Single-Image Super-Resolution: A Benchmark , 2014, ECCV.

[16]  Luc Van Gool,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Tingting Lv,et al.  Detecting Building Edges from High Spatial Resolution Remote Sensing Imagery Using Richer Convolution Features Network , 2018, Remote. Sens..

[19]  Jian Yang,et al.  MemNet: A Persistent Memory Network for Image Restoration , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[20]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Narendra Ahuja,et al.  Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Junjun Jiang,et al.  A Progressively Enhanced Network for Video Satellite Imagery Superresolution , 2018, IEEE Signal Processing Letters.

[24]  Lizhe Wang,et al.  SuperPCA: A Superpixelwise PCA Approach for Unsupervised Feature Extraction of Hyperspectral Imagery , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[25]  Yongjun Zhang,et al.  Large-Scale Remote Sensing Image Retrieval by Deep Hashing Neural Networks , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[26]  Junjun Jiang,et al.  FusionGAN: A generative adversarial network for infrared and visible image fusion , 2019, Inf. Fusion.

[27]  Jorge Núñez,et al.  Super-Resolution of Remotely Sensed Images With Variable-Pixel Linear Reconstruction , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Zhenfeng Shao,et al.  Remote Sensing Image Fusion With Deep Convolutional Neural Network , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[29]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[31]  Junmo Kim,et al.  A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[33]  Zhongyuan Wang,et al.  SeaShips: A Large-Scale Precisely Annotated Dataset for Ship Detection , 2018, IEEE Transactions on Multimedia.

[34]  Shuiping Gou,et al.  Remote Sensing Image Super-Resolution Reconstruction Based on Nonlocal Pairwise Dictionaries and Double Regularization , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[35]  Junjun Jiang,et al.  Guided Locality Preserving Feature Matching for Remote Sensing Image Registration , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Chih-Yuan Yang,et al.  Fast Direct Super-Resolution by Simple Functions , 2013, 2013 IEEE International Conference on Computer Vision.

[38]  Xianming Liu,et al.  Hyperspectral Image Classification in the Presence of Noisy Labels , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[39]  Zhongyuan Wang,et al.  Super-Resolution for “Jilin-1” Satellite Video Imagery via a Convolutional Network , 2018, Sensors.

[40]  Christian Osendorfer,et al.  Image Super-Resolution with Fast Approximate Convolutional Sparse Coding , 2014, ICONIP.

[41]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[42]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[43]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[44]  Shuyuan Yang,et al.  Novel Super Resolution Restoration of Remote Sensing Images Based on Compressive Sensing and Example Patches-Aided Dictionary Learning , 2011, 2011 International Workshop on Multi-Platform/Multi-Sensor Remote Sensing and Mapping.

[45]  Jiayi Ma,et al.  Hyperspectral Unmixing with Robust Collaborative Sparse Regression , 2016, Remote. Sens..

[46]  Renjie Liao,et al.  Detail-Revealing Deep Video Super-Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[47]  Weixing Zhang,et al.  Deep Convolutional Neural Networks for Automated Characterization of Arctic Ice-Wedge Polygons in Very High Spatial Resolution Aerial Imagery , 2018, Remote. Sens..

[48]  Zhuowen Tu,et al.  Robust Point Matching via Vector Field Consensus , 2014, IEEE Transactions on Image Processing.

[49]  Shuai Li,et al.  End-to-End Airport Detection in Remote Sensing Images Combining Cascade Region Proposal Networks and Multi-Threshold Detection Networks , 2018, Remote. Sens..

[50]  Jiayi Ma,et al.  Infrared and visible image fusion methods and applications: A survey , 2018, Inf. Fusion.

[51]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[52]  Chiman Kwan,et al.  Blind Quality Assessment of Fused WorldView-3 Images by Using the Combinations of Pansharpening and Hypersharpening Paradigms , 2017, IEEE Geoscience and Remote Sensing Letters.

[53]  Feng Li,et al.  Universal HMT based super resolution for remote sensing images , 2008, 2008 15th IEEE International Conference on Image Processing.

[54]  Yue Liu,et al.  Recurrent Knowledge Distillation , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[55]  Zhongyuan Wang,et al.  Video Satellite Imagery Super Resolution via Convolutional Neural Networks , 2017, IEEE Geoscience and Remote Sensing Letters.

[56]  Jitendra Malik,et al.  Cross Modal Distillation for Supervision Transfer , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Shawn D. Newsam,et al.  Learning Low Dimensional Convolutional Neural Networks for High-Resolution Remote Sensing Image Retrieval , 2016, Remote. Sens..

[58]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.