Aerial Spectral Super-Resolution using Conditional Adversarial Networks

Inferring spectral signatures from ground based natural images has acquired a lot of interest in applied deep learning. In contrast to the spectra of ground based images, aerial spectral images have low spatial resolution and suffer from higher noise interference. In this paper, we train a conditional adversarial network to learn an inverse mapping from a trichromatic space to 31 spectral bands within 400 to 700 nm. The network is trained on AeroCampus, a first of its kind aerial hyperspectral dataset. AeroCampus consists of high spatial resolution color images and low spatial resolution hyperspectral images (HSI). Color images synthesized from 31 spectral bands are used to train our network. With a baseline root mean square error of 2.48 on the synthesized RGB test data, we show that it is possible to generate spectral signatures in aerial imagery.

[1]  Konrad Schindler,et al.  Learned Spectral Super-Resolution , 2017, ArXiv.

[2]  Sabine Süsstrunk,et al.  What is the space of spectral sensitivity functions for digital color cameras? , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[3]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[4]  Ajmal S. Mian,et al.  Hierarchical Beta Process with Gaussian Process Prior for Hyperspectral Image Super Resolution , 2016, ECCV.

[5]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Shree K. Nayar,et al.  Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum , 2010, IEEE Transactions on Image Processing.

[7]  Wang Yingkuan.  Computer Vision Technology for Food Quality Evaluation , 2009 .

[8]  Angel Domingo Sappa,et al.  Infrared Image Colorization Based on a Triplet DCGAN Architecture , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  Radu Timofte,et al.  In Defense of Shallow Learned Spectral Reconstruction from RGB Images , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[10]  Dimitris N. Metaxas,et al.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[11]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[12]  Boaz Arad,et al.  Sparse Recovery of Hyperspectral Signal from Natural RGB Images , 2016, ECCV.

[13]  Joost van de Weijer,et al.  Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[14]  D. Mulla.  Twenty five years of remote sensing in precision agriculture: Key advances and remaining knowledge gaps , 2013 .

[15]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[16]  Yoshua Bengio,et al.  The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  Wei Huang,et al.  Detecting shadows in high-resolution remote-sensing images of urban areas using spectral and spatial features , 2015 .

[21]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[22]  Matthew J. Hoffman,et al.  Aerial Vehicle Tracking by Adaptive Fusion of Hyperspectral Likelihood Maps , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[23]  Dong Liu,et al.  HSCNN: CNN-Based Hyperspectral Image Recovery from Spectrally Undersampled Projections , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[24]  Luc Van Gool,et al.  A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution , 2014, ACCV.

[25]  Konrad Schindler,et al.  Hyperspectral Super-Resolution by Coupled Spectral Unmixing , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[26]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[28]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[29]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[30]  Michael S. Brown,et al.  Training-Based Spectral Reconstruction from a Single RGB Image , 2014, ECCV.

[31]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[32]  Shutao Li,et al.  Hyperspectral Image Super-Resolution via Non-local Sparse Tensor Factorization , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).