HyperPNN: Hyperspectral Pansharpening via Spectrally Predictive Convolutional Neural Networks

Hyperspectral (HS) pansharpening intends to synthesize a HS image with a registered panchromatic image, to generate an enhanced image with simultaneous high spectral resolution and high spatial resolution. However, the spectral range gap between the two kinds of images and the need to resolve details for many continuous narrow bands make the technique prone to spectral distortion and spatial blurring. To mitigate the problems, we propose a new HS pansharpening framework via spectrally predictive convolutional neural networks (HyperPNN). In our proposed HyperPNN, spectrally predictive structure is introduced to strengthen the spectral prediction capability of a pansharpening network. Following the concept of the proposed HyperPNN, two specific pansharpening convolutional neural network (CNN) models, i.e., HyperPNN1 and HyperPNN2, are designed. Experimental results from three datasets suggest the excellent performance of our CNN-based HS pansharpening methods.

[1]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[2]  Te-Ming Tu,et al.  A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[3]  Hassan Ghassemian,et al.  A review of remote sensing image fusion methods , 2016, Inf. Fusion.

[4]  Abhinav Vishnu,et al.  Deep learning for computational chemistry , 2017, J. Comput. Chem..

[5]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[6]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[7]  V. K. Shettigara,et al.  A generalized component substitution technique for spatial enhancement of multispectral images using , 1992 .

[8]  Liangpei Zhang,et al.  A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[9]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[10]  Paul D. Gader,et al.  Signal and Image Processing in Hyperspectral Remote Sensing [From the Guest Editors] , 2014 .

[11]  L. Wald,et al.  Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images , 1997 .

[12]  A. Mohammadzadeh,et al.  Road extraction based on fuzzy logic and mathematical morphology from pan‐sharpened ikonos images , 2006 .

[13]  Naoto Yokoya,et al.  Coupled Nonnegative Matrix Factorization Unmixing for Hyperspectral and Multispectral Data Fusion , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Lei Zhang,et al.  Exploiting Clustering Manifold Structure for Hyperspectral Imagery Super-Resolution , 2018, IEEE Transactions on Image Processing.

[15]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[16]  Jocelyn Chanussot,et al.  A Regression-Based High-Pass Modulation Pansharpening Approach , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[17]  Aleksandra Pizurica,et al.  Processing of Multiresolution Thermal Hyperspectral and Digital Color Data: Outcome of the 2014 IEEE GRSS Data Fusion Contest , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[18]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[19]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[22]  Jocelyn Chanussot,et al.  A Convex Formulation for Hyperspectral Image Superresolution via Subspace-Based Regularization , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[23]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Jon Atli Benediktsson,et al.  Advances in Hyperspectral Image Classification: Earth Monitoring with Statistical Learning Methods , 2013, IEEE Signal Processing Magazine.

[25]  Davide Cozzolino,et al.  Target-Adaptive CNN-Based Pansharpening , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[26]  Yunsong Li,et al.  Hyperspectral Pansharpening With Guided Filter , 2017, IEEE Geoscience and Remote Sensing Letters.

[27]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[28]  Simon J. Godsill,et al.  R-FUSE: Robust Fast Fusion of Multiband Images Based on Solving a Sylvester Equation , 2016, IEEE Signal Processing Letters.

[29]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation : a spectral preserve image fusion technique for improving spatial details , 2001 .

[30]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[31]  Jocelyn Chanussot,et al.  Image fusion and spectral unmixing of hyperspectral images for spatial improvement of classification maps , 2012, 2012 IEEE International Geoscience and Remote Sensing Symposium.

[32]  Te-Ming Tu,et al.  A new look at IHS-like image fusion methods , 2001, Inf. Fusion.

[33]  Jean-Yves Tourneret,et al.  Bayesian Fusion of Multi-Band Images , 2013, IEEE Journal of Selected Topics in Signal Processing.