Remote Sensing Image Fusion With Deep Convolutional Neural Network

Remote sensing images with different spatial and spectral resolution, such as panchromatic (PAN) images and multispectral (MS) images, can be captured by many earth-observing satellites. Normally, PAN images possess high spatial resolution but low spectral resolution, while MS images have high spectral resolution with low spatial resolution. In order to integrate spatial and spectral information contained in the PAN and MS images, image fusion techniques are commonly adopted to generate remote sensing images at both high spatial and spectral resolution. In this study, based on the deep convolutional neural network, a remote sensing image fusion method that can adequately extract spectral and spatial features from source images is proposed. The major innovation of this study is that the proposed fusion method contains a two branches network with the deeper structure which can capture salient features of the MS and PAN images separately. Besides, the residual learning is adopted in our network to thoroughly study the relationship between the high- and low-resolution MS images. The proposed method mainly consists of two procedures. First, spatial and spectral features are respectively extracted from the MS and PAN images by convolutional layers with different depth. Second, the feature fusion procedure utilizes the extracted features from the former step to yield fused images. By evaluating the performance on the QuickBird and Gaofen-1 images, our proposed method provides better results compared with other classical methods.

[1]  T. J. Majumdar,et al.  ERS-2 SAR and IRS-1C LISS III data fusion: A PCA approach to improve remote sensing based geological interpretation , 2007 .

[2]  Gonzalo Pajares Martinsanz,et al.  A wavelet-based image fusion tutorial , 2004 .

[3]  Jun Liu,et al.  Human visual system consistent quality assessment for remote sensing image fusion , 2015 .

[4]  Jia Liu,et al.  Change detection based on deep feature representation and mapping transformation for multi-spatial-resolution remote sensing images , 2016 .

[5]  Qingshan Liu,et al.  Learning Multiscale Deep Features for High-Resolution Satellite Image Scene Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[6]  Neil Genzlinger.  A. and Q , 2006 .

[7]  Lucien Wald,et al.  Quality of high resolution synthesised images: Is there a simple criterion ? , 2000 .

[8]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[9]  Shutao Li,et al.  Multifocus Image Fusion and Restoration With Sparse Representation , 2010, IEEE Transactions on Instrumentation and Measurement.

[10]  Yann LeCun,et al.  Convolutional neural networks applied to house numbers digit classification , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[11]  Brian P. Salmon,et al.  Multiview Deep Learning for Land-Use Classification , 2015, IEEE Geoscience and Remote Sensing Letters.

[12]  Ting Liu,et al.  Remote sensing image fusion via wavelet transform and sparse representation , 2015 .

[13]  B. S. Manjunath,et al.  Multisensor Image Fusion Using the Wavelet Transform , 1995, CVGIP Graph. Model. Image Process..

[14]  Trevor Darrell,et al.  Part-Based R-CNNs for Fine-Grained Category Detection , 2014, ECCV.

[15]  S. Baronti,et al.  Multispectral and panchromatic data fusion assessment without reference , 2008 .

[16]  Mohamed-Jalal Fadili,et al.  Wavelets, Ridgelets, and Curvelets for Poisson Noise Removal , 2008, IEEE Transactions on Image Processing.

[17]  Yonghyun Kim,et al.  Improved Additive-Wavelet Image Fusion , 2011, IEEE Geoscience and Remote Sensing Letters.

[18]  Andrea Garzelli,et al.  Pansharpening of Multispectral Images Based on Nonlocal Parameter Optimization , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[19]  Gang Wang,et al.  Deep Learning-Based Classification of Hyperspectral Data , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[20]  Jocelyn Chanussot,et al.  Comparison of Pansharpening Algorithms: Outcome of the 2006 GRS-S Data-Fusion Contest , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[21]  丁萌,et al.  Research on fusion method for infrared and visible images via compressive sensing , 2013 .

[22]  Gerlind Plonka-Hoch,et al.  Computing with Curvelets: From Image Processing to Turbulent Flows , 2009, Computing in Science & Engineering.

[23]  Te-Ming Tu,et al.  A new look at IHS-like image fusion methods , 2001, Inf. Fusion.

[24]  Xiuping Jia,et al.  Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[25]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[27]  Jinying Zhong,et al.  Remote Sensing Image Fusion with Convolutional Neural Network , 2016 .

[28]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[29]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[30]  W. Marsden.  I and J , 2012 .

[31]  Xiaogang Wang,et al.  DeepID-Net: Deformable deep convolutional neural networks for object detection , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Nicholas C. Coops,et al.  Multi-scale analysis of relationship between imperviousness and urban tree height using airborne remote sensing , 2017 .

[33]  Shutao Li,et al.  Pixel-level image fusion with simultaneous orthogonal matching pursuit , 2012, Inf. Fusion.

[34]  R. Pu,et al.  Remotely Sensing Image Fusion Based on Wavelet Transform and Human Vision System , 2015 .

[35]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Wen Dou,et al.  A general framework for component substitution image fusion: An implementation using the fast image fusion method , 2007, Comput. Geosci..

[37]  Xiaogang Wang,et al.  Joint Deep Learning for Pedestrian Detection , 2013, 2013 IEEE International Conference on Computer Vision.

[38]  L. Wald,et al.  Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images , 1997 .

[39]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[40]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[41]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[42]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Shutao Li,et al.  Remote Sensing Image Fusion via Sparse Representations Over Learned Dictionaries , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[44]  Qian Du,et al.  Hyperspectral Image Classification Using Deep Pixel-Pair Features , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[45]  Jean-Yves Tourneret,et al.  Hyperspectral and Multispectral Image Fusion Based on a Sparse Representation , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[46]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[47]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[48]  Lei Cao,et al.  A GIHS-based spectral preservation fusion method for remote sensing images using edge restored spectral modulation , 2014 .

[49]  Minh N. Do,et al.  The Nonsubsampled Contourlet Transform: Theory, Design, and Applications , 2006, IEEE Transactions on Image Processing.

[50]  Johannes R. Sveinsson,et al.  Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[51]  Te-Ming Tu,et al.  A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[52]  Baohua Zhang,et al.  A fusion algorithm for infrared and visible images based on saliency analysis and non-subsampled Shearlet transform , 2015 .

[53]  Michael Möller,et al.  An Adaptive IHS Pan-Sharpening Method , 2010, IEEE Geoscience and Remote Sensing Letters.

[54]  B. S. Manjunath,et al.  Multi-sensor image fusion using the wavelet transform , 1994, Proceedings of 1st International Conference on Image Processing.

[55]  Juha Hyyppä,et al.  Object-based analysis of multispectral airborne laser scanner data for land cover classification and map updating , 2017 .

[56]  Jörg Trentmann,et al.  Remote sensing of solar surface radiation for climate monitoring — the CM-SAF retrieval in international comparison , 2012 .