Paradigm shifts in super-resolution techniques for remote sensing applications

Super-resolution (SR) algorithms have now become a bottleneck for several remote sensing applications. SR is a technique that enhances minute details of the image by increasing spatial resolution of imaging systems. SR overcomes the problems of conventional resolution enhancement techniques such as introduction of noise, spectral distortion, and lack of clarity in the details of the image. In this paper, a survey has been conducted since the inception of SR algorithm till the latest state-of-the-art SR techniques to elucidate the importance of the SR algorithms that lead to paradigm shifts in the last two decades revolutionizing toward visually pleasing high-resolution image. Inspired from the natural images, the algorithms addressing the SR problems such as ill-posed, prior and regularization problem, inverse problem, multi-frame problem and illumination and shadow problem in remote sensing applications are analyzed. For an intuitive understanding of the paradigm shifts, publicly available images are tested with representative paradigm shift SR algorithms. The result of this paradigm shift analysis is done both qualitatively and quantitatively in terms of blurs in the image, pattern clarity, edge strength, and super-resolving capability. The convergence of the natural image to the remote sensed image is critically analyzed. The challenges with possible solutions for super-resolving the remote sensed image are recommended. On experimentation, it is found that deep learning-based SR algorithms produces visually pleasing images retaining sharp edges, enhanced spatial data, and clarity in feature representation while zooming at a certain level beyond interest.

[1]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Di Zhang,et al.  Morphable model space based face super-resolution reconstruction and recognition , 2012, Image Vis. Comput..

[3]  Subhasis Chaudhuri,et al.  Single frame image super-resolution: should we process locally or globally? , 2007, Multidimens. Syst. Signal Process..

[4]  Daniel Cremers,et al.  Fight Ill-Posedness with Ill-Posedness: Single-shot Variational Depth Super-Resolution from Shading , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[5]  Yi Wang,et al.  Super-resolution mosaicking of UAV surveillance video , 2008, 2008 15th IEEE International Conference on Image Processing.

[6]  Rasim Latifovic,et al.  Landsat Super-Resolution Enhancement Using Convolution Neural Networks and Sentinel-2 for Training , 2018, Remote. Sens..

[7]  Liu Pin,et al.  Super-resolution Reconstruction of Satellite Video Images Based on Interpolation Method , 2017 .

[8]  Peter Meer,et al.  From a Robust Hierarchy to a Hierarchy of Robustness , 2001 .

[9]  Sudhish N. George,et al.  Single image super resolution using neighbor embedding and statistical prediction model , 2017, Comput. Electr. Eng..

[10]  Thomas S. Huang,et al.  Multiframe image restoration and registration , 1984 .

[11]  Francisco de Borja Rodríguez Ortiz,et al.  A two-step neural-network based algorithm for fast image super-resolution , 2007, Image Vis. Comput..

[12]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[13]  Masatoshi Okutomi,et al.  Direct super-resolution and registration using raw CFA images , 2004, CVPR 2004.

[14]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Rynson W. H. Lau,et al.  Efficient image super-resolution integration , 2018, The Visual Computer.

[16]  Xu-Cheng Yin,et al.  Satellite super-resolution images depending on deep learning methods: A comparative study , 2017, 2017 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC).

[17]  Jie Yang,et al.  Local spatial information for image super-resolution , 2018, Cognitive Systems Research.

[18]  Nirmal K. Bose,et al.  Recursive implementation of total least squares algorithm for image reconstruction from noisy, undersampled multiframes , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[19]  A. Vasuki,et al.  A Novel approach to super resolution image reconstruction algorithm from low resolution panchromatic images , 2015, 2015 3rd International Conference on Signal Processing, Communication and Networking (ICSCN).

[20]  D. Yeung,et al.  Super-resolution through neighbor embedding , 2004, CVPR 2004.

[21]  Umesh Ghanekar,et al.  Classification of priors and regularization techniques appurtenant to single image super-resolution , 2019, The Visual Computer.

[22]  Zhenwei Shi,et al.  Super-Resolution for Remote Sensing Images via Local–Global Combined Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[23]  Shuvra S. Bhattacharyya,et al.  Multi-Frame Super Resolution with Deep Residual Learning on Flow Registered Non-Integer Pixel Images , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[24]  Francesca Bovolo,et al.  Single Image Super-Resolution for Optical Satellite Scenes Using Deep Deconvolutional Network , 2019, ICIAP.

[25]  Graham W. Taylor,et al.  Adaptive deconvolutional networks for mid and high level feature learning , 2011, 2011 International Conference on Computer Vision.

[26]  Qijun Zhao,et al.  Deepside: A general deep framework for salient object detection , 2019, Neurocomputing.

[27]  M. Körner,et al.  SINGLE-IMAGE SUPER RESOLUTION FOR MULTISPECTRAL REMOTE SENSING DATA USING CONVOLUTIONAL NEURAL NETWORKS , 2016 .

[28]  P. Dare.  Shadow Analysis in High-Resolution Satellite Imagery of Urban Areas , 2005 .

[29]  Hairong Qi,et al.  Aerial Image Super Resolution via Wavelet Multiscale Convolutional Neural Networks , 2018, IEEE Geoscience and Remote Sensing Letters.

[30]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Francisco de Borja Rodríguez Ortiz,et al.  Accurate and Robust Image Superresolution by Neural Processing of Local Image Representations , 2005, ICANN.

[33]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[34]  Nikolas P. Galatsanos,et al.  Reconstruction of a High Resolution Image from Multiple Low Resolution Images , 2002 .

[35]  Jie Wu,et al.  Multiple-image super resolution using both reconstruction optimization and deep neural network , 2017, 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[36]  Yuan Zhou,et al.  Depth image super-resolution based on joint sparse coding , 2020, Pattern Recognit. Lett..

[37]  B. Gunturk,et al.  Multiframe resolution-enhancement methods for compressed video , 2002, IEEE Signal Processing Letters.

[38]  Ramón M. Rodríguez-Dagnino.  A Super-Resolution Image Reconstruction using Natural Neighbor Interpolation , 2015, Computación y Sistemas.

[39]  Wei Wang,et al.  Super-Resolution Reconstruction of High-Resolution Satellite ZY-3 TLC Images , 2017, Sensors.

[40]  Nirmal K. Bose,et al.  Recursive reconstruction of high resolution image from noisy undersampled multiframes , 1990, IEEE Trans. Acoust. Speech Signal Process..

[41]  Anat Levin,et al.  Accurate Blur Models vs. Image Priors in Single Image Super-resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[42]  Yi Sun,et al.  A deep learning method for image super-resolution based on geometric similarity , 2019, Signal Process. Image Commun..

[43]  R. Barnard,et al.  A practical enhanced-resolution integrated optical-digital imaging camera (PERIODIC) , 2009, Defense + Commercial Sensing.

[44]  Adam Van Etten,et al.  The Effects of Super-Resolution on Object Detection Performance in Satellite Imagery , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[45]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[46]  Peyman Milanfar,et al.  Super-resolution imaging , 2011 .

[47]  Jiangjiang Liu,et al.  Salient Objects in Clutter: Bringing Salient Object Detection to the Foreground , 2018, ECCV.

[48]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Elif Sertel,et al.  Single-frame super resolution of remote-sensing images by convolutional neural networks , 2018 .

[50]  Sri Addepalli,et al.  Global motion based video super-resolution reconstruction using discrete wavelet transform , 2018, Multimedia Tools and Applications.

[51]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[52]  Yanbing Xue,et al.  Super-Resolution Image Reconstruction Based on K-Means-Markov Network , 2009, 2009 Fifth International Conference on Natural Computation.

[53]  Hongyi Liu,et al.  A New Pan-Sharpening Method With Deep Neural Networks , 2015, IEEE Geoscience and Remote Sensing Letters.

[54]  Zongxu Pan,et al.  Super-Resolution of Remote Sensing Images Based on Transferred Generative Adversarial Network , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[55]  Yuan Shuai,et al.  Accurate Image Super-Resolution Using Cascaded Multi-Column Convolutional Neural Networks , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[56]  David O. Walsh,et al.  Direct method for superresolution , 1994 .

[57]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Zhongyuan Wang,et al.  Video Satellite Imagery Super Resolution via Convolutional Neural Networks , 2017, IEEE Geoscience and Remote Sensing Letters.

[59]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[60]  Liangpei Zhang,et al.  Multiframe Super-Resolution Employing a Spatially Weighted Total Variation Model , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[61]  Sudhish N. George,et al.  Robust single image super resolution using neighbor embedding and fusion in wavelet domain , 2017, Comput. Electr. Eng..

[62]  Encyclopedia of Multimedia, 2nd Ed , 2008 .

[63]  LiJie,et al.  Image super-resolution , 2016 .

[64]  Shmuel Peleg,et al.  Image sequence enhancement using sub-pixel displacements , 1988, Proceedings CVPR '88: The Computer Society Conference on Computer Vision and Pattern Recognition.

[65]  Qian Du,et al.  Remote sensing images super-resolution with deep convolution networks , 2019, Multimedia Tools and Applications.

[66]  Junjun Jiang,et al.  Satellite Image Super-Resolution via Multi-Scale Residual Deep Neural Network , 2019, Remote. Sens..

[67]  Feilong Cao,et al.  Super-resolution using neighbourhood regression with local structure prior , 2019, Signal Process. Image Commun..

[68]  Truong Q. Nguyen,et al.  Maximum-likelihood parameter estimation for image ringing-artifact removal , 2001, IEEE Trans. Circuits Syst. Video Technol..

[69]  Yihong Gong,et al.  Image and Video Super Resolution Techniques , 2008, Encyclopedia of Multimedia.

[70]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[71]  Khizar Hayat,et al.  Super-Resolution via Deep Learning , 2017, Digit. Signal Process..

[72]  Yang Cao,et al.  Contrast Prior and Fluid Pyramid Integration for RGBD Salient Object Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  Christine Pohl,et al.  Multisensor image fusion in remote sensing: concepts, methods and applications , 1998 .

[74]  Panagiotis Tsakalides,et al.  Survey of Deep-Learning Approaches for Remote Sensing Observation Enhancement , 2019, Sensors.

[75]  H Stark,et al.  High-resolution image recovery from image-plane arrays, using convex projections. , 1989, Journal of the Optical Society of America. A, Optics and image science.

[76]  R. Gerchberg.  Super-resolution through Error Energy Reduction , 1974 .

[77]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[78]  Yang Xu,et al.  Pan-Sharpening Based on Multilevel Coupled Deep Network , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[79]  Chih-Yuan Yang,et al.  Single-Image Super-Resolution: A Benchmark , 2014, ECCV.

[80]  Li Meng,et al.  Subpixel Motion Estimation for Super-Resolution Image Sequence Enhancement , 1998, J. Vis. Commun. Image Represent..

[81]  Hassan Ghassemian,et al.  A new pansharpening method using multi resolution analysis framework and deep neural networks , 2017, 2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA).

[82]  Dario Izzo,et al.  Super-resolution of PROBA-V images using convolutional neural networks , 2019, Astrodynamics.

[83]  Narendra Ahuja,et al.  Super-resolving Noisy Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[84]  Liangpei Zhang,et al.  A super-resolution reconstruction algorithm for hyperspectral images , 2012, Signal Process..

[85]  G. Rohith,et al.  Performance analysis of Satellite Image Super Resolution using Deep Learning Techniques , 2019, 2019 IEEE Bombay Section Signature Conference (IBSSC).

[86]  Zongxu Pan,et al.  Achieving Super-Resolution Remote Sensing Images via the Wavelet Transform Combined With the Recursive Res-Net , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[87]  Cemal Köse,et al.  A novel image decomposition-based hybrid technique with super-resolution method for multi-focus image fusion , 2019, Inf. Fusion.

[88]  Masashi Watanabe,et al.  Super-resolution through non-linear enhancement filters , 2013, 2013 IEEE International Conference on Image Processing.

[89]  Xing Xu,et al.  Sparse representation based image super-resolution on the KNN based dictionaries , 2018 .

[90]  Kaushal K. Shukla,et al.  Efficient Algorithms for Discrete Wavelet Transform: With Applications to Denoising and Fuzzy Inference Systems , 2013 .

[91]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[92]  Jian Li,et al.  Super resolution SAR imaging via parametric spectral estimation methods , 1999 .

[93]  Hongwei Xie,et al.  Study of Supper Resolution Processing Methods for Thick Pinhole Image , 2013 .

[94]  Weikuan Jia,et al.  Single image super-resolution using a polymorphic parallel CNN , 2018, Applied Intelligence.

[95]  Gholamreza Anbarjafari,et al.  IMAGE Resolution Enhancement by Using Discrete and Stationary Wavelet Decomposition , 2011, IEEE Transactions on Image Processing.

[96]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[97]  Kin-Man Lam,et al.  Image Super-resolution via Feature-augmented Random Forest , 2017, Signal Process. Image Commun..

[98]  Sumohana S. Channappayya,et al.  Blind image quality evaluation using perception based features , 2015, 2015 Twenty First National Conference on Communications (NCC).

[99]  Thomas B. Moeslund,et al.  Super-resolution: a comprehensive survey , 2014, Machine Vision and Applications.

[100]  Alan C. Bovik,et al.  No-Reference Image Quality Assessment in the Spatial Domain , 2012, IEEE Transactions on Image Processing.

[101]  Peyman Milanfar,et al.  Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement , 2001, IEEE Trans. Image Process..

[102]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[103]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[104]  Feng Li,et al.  Universal HMT based super resolution for remote sensing images , 2008, 2008 15th IEEE International Conference on Image Processing.

[105]  Qijun Zhao,et al.  Refinet: A Deep Segmentation Assisted Refinement Network for Salient Object Detection , 2019, IEEE Transactions on Multimedia.

[106]  Yang Wang,et al.  High Quality Remote Sensing Image Super-Resolution Using Deep Memory Connected Network , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[107]  Neeraj Kumar,et al.  Convolutional neural networks for wavelet domain super resolution , 2017, Pattern Recognit. Lett..

[108]  Thomas S. Huang,et al.  Coupled Dictionary Training for Image Super-Resolution , 2012, IEEE Transactions on Image Processing.

[109]  Thomas S. Huang,et al.  Learning-Based Nonparametric Image Super-Resolution , 2006, EURASIP J. Adv. Signal Process..

[110]  Michael Elad,et al.  A Statistical Prediction Model Based on Sparse Representations for Single Image Super-Resolution , 2014, IEEE Transactions on Image Processing.

[111]  Gunamani Jena,et al.  POCS Based Super-Resolution Image Reconstruction Using an Adaptive Regularization Parameter , 2011, ArXiv.

[112]  Kaushal K. Shukla,et al.  Efficient Algorithms for Discrete Wavelet Transform , 2013, SpringerBriefs in Computer Science.

[113]  Sridha Sridharan,et al.  Super-resolution for biometrics: A comprehensive survey , 2018, Pattern Recognit..

[114]  Jing Zhang,et al.  Multi-view Sparsity Preserving Projection for dimension reduction , 2016, Neurocomputing.

[115]  Junjie Liu,et al.  Single-Image Super-Resolution for Remote Sensing Data Using Deep Residual-Learning Neural Network , 2017, ICONIP.

[116]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[117]  Takeo Kanade,et al.  Limits on Super-Resolution and How to Break Them , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[118]  Luca Maria Gambardella,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Flexible, High Performance Convolutional Neural Networks for Image Classification , 2022 .

[119]  Manjunath V. Joshi,et al.  Zoom Based Super-Resolution: A Fast Approach Using Particle Swarm Optimization , 2010, ICISP.

[120]  Yo-Sung Ho,et al.  Sparse Representation for No-Reference Quality Assessment of Satellite Stereo Images , 2019, IEEE Access.

[121]  Makoto Takizawa,et al.  Future Data and Security Engineering , 2014, Lecture Notes in Computer Science.

[122]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[123]  Lina J. Karam,et al.  Efficient Super-Resolution driven by saliency selectivity , 2011, 2011 18th IEEE International Conference on Image Processing.

[124]  Nasser Kehtarnavaz,et al.  Convolutional Autoencoder-Based Multispectral Image Fusion , 2019, IEEE Access.

[125]  Xin Zhang,et al.  Multi-example feature-constrained back-projection method for image super-resolution , 2016, Computational Visual Media.

[126]  William T. Freeman,et al.  Learning to Estimate Scenes from Images , 1998, NIPS.

[127]  Mohiy Hadhoud,et al.  Image Super-Resolution and Applications , 2012 .

[128]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[129]  William T. Freeman,et al.  Learning low-level vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[130]  Aggelos K. Katsaggelos,et al.  Parameter Estimation in Bayesian Reconstruction of Multispectral Images using Super Resolution Techniques , 2006, 2006 International Conference on Image Processing.

[131]  Peter Cheeseman,et al.  Super-Resolved Surface Reconstruction from Multiple Images , 1996 .

[132]  Shmuel Peleg,et al.  Improving image resolution using subpixel motion , 1987, Pattern Recognit. Lett..

[133]  Zhenghua Zhou,et al.  Single image super-resolution based on adaptive convolutional sparse coding and convolutional neural networks , 2019, J. Vis. Commun. Image Represent..

[134]  Konrad Schindler,et al.  Super-Resolution of Sentinel-2 Images: Learning a Globally Applicable Deep Neural Network , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[135]  Zongxu Pan,et al.  Super-Resolution of Single Remote Sensing Image Based on Residual Dense Backprojection Networks , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[136]  Cesar Beltran-Castanon,et al.  Super Resolution Approach Using Generative Adversarial Network Models for Improving Satellite Image Resolution , 2019, SIMBig.

[137]  Hugo Larochelle,et al.  Efficient Learning of Deep Boltzmann Machines , 2010, AISTATS.

[138]  Filiberto Pla,et al.  A New Deep Generative Network for Unsupervised Remote Sensing Single-Image Super-Resolution , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[139]  Hugh G. Lewis,et al.  Super-resolution target identification from remotely sensed images using a Hopfield neural network , 2001, IEEE Trans. Geosci. Remote. Sens..

[140]  Jie Li,et al.  Image super-resolution: The techniques, applications, and future , 2016, Signal Process..

[141]  Franco Gori,et al.  On an Iterative Method for Super-resolution , 1975 .

[142]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[143]  J. Ortiz,et al.  DEM shading method for the correction of pseudoscopic effect on multi-platform satellite imagery , 2014 .

[144]  Mudar Sarem,et al.  A novel multi-image super-resolution reconstruction method using anisotropic fractional order adaptive norm , 2014, The Visual Computer.

[145]  Peyman Milanfar,et al.  A computationally efficient superresolution image reconstruction algorithm , 2001, IEEE Trans. Image Process..

[146]  Xinying Xu,et al.  Deep Learning Based Single Image Super-resolution: A Survey , 2019, International Journal of Automation and Computing.

[147]  Sudhish N. George,et al.  Modified dictionary learning method for sparsity based single image super-resolution , 2016, 2016 3rd International Conference on Recent Advances in Information Technology (RAIT).

[148]  Junjun Jiang,et al.  Edge-Enhanced GAN for Remote Sensing Image Superresolution , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[149]  I. Shamsudin,et al.  A review of uses of satellite imagery in monitoring mangrove forests , 2016 .

[150]  Robert L. Stevenson,et al.  A Bayesian approach to image expansion for improved definitio , 1994, IEEE Trans. Image Process..