Panchromatic and multi-spectral image fusion for new satellites based on multi-channel deep model

With the launch and rapid development of new satellites such as WorldView-3, the bands number of multi-spectral images from new satellites is greatly increased. However, the spectral matching between the panchromatic image and multi-spectral images is deteriorated with the existing image fusion methods. In this paper, a novel method based on the multi-channel deep model is proposed to fuse images for new satellites. The deep model is implemented by convolutional neural networks and trained on each band to reduce the impact of spectral range mismatch. The proposed method also preserves the detailed information in multi-spectral images, which is ignored by the traditional methods. It also effectively alleviates the inconvenience for obtaining the remote sensing images by the data augmentation processing. In addition, it reduces the randomness of manual setting parameters using the parameter self-learning. Visual and quantitative assessments of fusion results show that the proposed method clearly improves the fusion quality compared to the state-of-the-art methods.

[1]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Andrea Garzelli,et al.  Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis , 2002, IEEE Trans. Geosci. Remote. Sens..

[3]  Shuyuan Yang,et al.  Novel Adaptive Component-Substitution-Based Pan-Sharpening Using Particle Swarm Optimization , 2015, IEEE Geoscience and Remote Sensing Letters.

[4]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[5]  Mufit Cetin,et al.  Intensity–hue–saturation-based image fusion using iterative linear regression , 2016 .

[6]  Jocelyn Chanussot,et al.  Multi-resolution analysis techniques and nonlinear PCA for hybrid pansharpening applications , 2016, Multidimens. Syst. Signal Process..

[7]  Zhou Yu,et al.  Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Haipeng Wang,et al.  Application of deep-learning algorithms to MSTAR data , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[9]  Hua Yang,et al.  Sparse Feature Fidelity for Perceptual Image Quality Assessment , 2013, IEEE Transactions on Image Processing.

[10]  Onisimo Mutanga,et al.  Testing the capabilities of the new WorldView-3 space-borne sensor’s red-edge spectral band in discriminating and mapping complex grassland management treatments , 2017 .

[11]  Shutao Li,et al.  Pixel-level image fusion: A survey of the state of the art , 2017, Inf. Fusion.

[12]  Xavier Otazu,et al.  Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[13]  Andrea Garzelli,et al.  Interband structure modeling for Pan-sharpening of very high-resolution multispectral images , 2005, Inf. Fusion.

[14]  Chen Mi.  Remote Sensing Image Fusion Based on Contourlet Transform , 2006 .

[15]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[16]  Luisa Verdoliva,et al.  Land Use Classification in Remote Sensing Images by Convolutional Neural Networks , 2015, ArXiv.

[17]  Liangpei Zhang,et al.  A Practical Compressed Sensing-Based Pan-Sharpening Method , 2012, IEEE Geoscience and Remote Sensing Letters.

[18]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Hongyi Liu,et al.  A New Pan-Sharpening Method With Deep Neural Networks , 2015, IEEE Geoscience and Remote Sensing Letters.

[20]  Shutao Li,et al.  A New Pan-Sharpening Method Using a Compressed Sensing Technique , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[21]  Abdenour Hadid,et al.  Deep convolutional hashing using pairwise multi-label supervision for large-scale visual search , 2017, Signal Process. Image Commun..

[22]  Saygin Abdikan,et al.  Evaluation of image fusion methods using PALSAR, RADARSAT-1 and SPOT images for land use/ land cover classification , 2016, Journal of the Indian Society of Remote Sensing.

[23]  Zhou Yu,et al.  Hashing with List-Wise learning to rank , 2014, SIGIR.

[24]  Jinying Zhong,et al.  Remote Sensing Image Fusion with Convolutional Neural Network , 2016 .

[25]  Qing Guo,et al.  Ehlers pan-sharpening performance enhancement using HCS transform for n-band data sets , 2017 .

[26]  Hongwei Liu,et al.  Convolutional Neural Network With Data Augmentation for SAR Target Recognition , 2016, IEEE Geoscience and Remote Sensing Letters.

[27]  Jagalingam Pushparaj,et al.  Comparison of various pan-sharpening methods using Quickbird-2 and Landsat-8 imagery , 2017, Arabian Journal of Geosciences.

[28]  V. Sowmya,et al.  Application of M-band wavelet in pan-sharpening , 2017, J. Intell. Fuzzy Syst..

[29]  Zhang Xiao-dong.  Fusion algorithm of remote sensing images based on nonsubsampled pyramid and empirical mode demoposition , 2012 .

[30]  Svetlana Lazebnik,et al.  Multi-scale Orderless Pooling of Deep Convolutional Activation Features , 2014, ECCV.

[31]  Pengfei Liu,et al.  A New Geometry Enforcing Variational Model for Pan-Sharpening , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[32]  Gustau Camps-Valls,et al.  Measuring the Spatial and Spectral Performance of WorldView-3 , 2015 .

[33]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.