Super-resolution of Omnidirectional Images Using Adversarial Learning

An omnidirectional image (ODI) enables viewers to look in every direction from a fixed point through a head-mounted display providing an immersive experience compared to that of a standard image. Designing immersive virtual reality systems with ODIs is challenging as they require high resolution content. In this paper, we study super-resolution for ODIs and propose an improved generative adversarial network based model which is optimized to handle the artifacts obtained in the spherical observational space. Specifically, we propose to use a fast PatchGAN discriminator, as it needs fewer parameters and improves the super-resolution at a fine scale. We also explore the generative models with adversarial learning by introducing a spherical-content specific loss function, called 360-SS. To train and test the performance of our proposed model we prepare a dataset of 4500 ODIs. Our results demonstrate the efficacy of the proposed method and identify new challenges in ODI super-resolution for future investigations.

[1]  Julián Cabrera,et al.  Voronoi-based Objective Quality Metrics for Omnidirectional Video , 2019, 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX).

[2]  Pascal Frossard,et al.  Plenoptic based super-resolution for omnidirectional image sequences , 2010, 2010 IEEE International Conference on Image Processing.

[3]  Louis B. Rall,et al.  Automatic differentiation , 1981 .

[4]  Giuseppe Valenzise,et al.  Learning-based tone mapping operator for image matching , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[5]  Mei Yu,et al.  Weighted-to-Spherically-Uniform SSIM Objective Quality Evaluation for Panoramic Video , 2018, 2018 14th IEEE International Conference on Signal Processing (ICSP).

[6]  Pascal Frossard,et al.  Joint Registration and Super-Resolution With Omnidirectional Images , 2011, IEEE Transactions on Image Processing.

[7]  Aakanksha Rana,et al.  Graph-cut-based model for spectral-spatial classification of hyperspectral images , 2014, 2014 IEEE Geoscience and Remote Sensing Symposium.

[8]  Steven C. H. Hoi,et al.  Deep Learning for Image Super-Resolution: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Patrick Pérez,et al.  Feature Learning for the Image Retrieval Task , 2014, ACCV Workshops.

[10]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Mehdi Bennis,et al.  Toward Low-Latency and Ultra-Reliable Virtual Reality , 2018, IEEE Network.

[12]  Giuseppe Valenzise,et al.  Learning-Based Tone Mapping Operator for Efficient Image Matching , 2019, IEEE Transactions on Multimedia.

[13]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Giuseppe Valenzise,et al.  Learning-based adaptive tone mapping for keypoint detection , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[15]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[16]  Alexia Jolicoeur-Martineau,et al.  The relativistic discriminator: a key element missing from standard GAN , 2018, ICLR.

[17]  Aljoscha Smolic,et al.  Viewport-aware adaptive 360° video streaming using tiles for virtual reality , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[18]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[19]  Giuseppe Valenzise,et al.  An evaluation of HDR image matching under extreme illumination changes , 2016, 2016 Visual Communications and Image Processing (VCIP).

[20]  Cagri Ozcinar,et al.  Towards Generating Ambisonics Using Audio-visual Cue for Virtual Reality , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[21]  Catarina Brites,et al.  Saliency-driven omnidirectional imaging adaptive coding: Modeling and assessment , 2017, 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP).

[22]  Giuseppe Valenzise,et al.  Optimizing tone mapping operators for keypoint detection under illumination changes , 2016, 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP).

[23]  Lu Yu,et al.  Weighted-to-Spherically-Uniform Quality Evaluation for Omnidirectional Video , 2017, IEEE Signal Processing Letters.

[24]  Emin Zerman,et al.  Colornet - Estimating Colorfulness in Natural Images , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[25]  Zhenzhong Chen,et al.  Subjective Panoramic Video Quality Assessment Database for Coding Applications , 2018, IEEE Transactions on Broadcasting.

[26]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[27]  Krista A. Ehinger,et al.  Recognizing scene viewpoint using panoramic place representation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[29]  Wei Sun,et al.  A Large-Scale Compressed 360-Degree Spherical Image Database: From Subjective Quality Evaluation to Objective Model Comparison , 2018, 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP).

[30]  Radu Timofte,et al.  2018 PIRM Challenge on Perceptual Image Super-resolution , 2018, ArXiv.

[31]  Cagri Ozcinar,et al.  Visual Attention-Aware Omnidirectional Video Streaming Using Optimal Tiles for Virtual Reality , 2019, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[32]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[33]  Wangmeng Zuo,et al.  Learning a Single Convolutional Super-Resolution Network for Multiple Degradations , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Thomas B. Moeslund,et al.  Super-resolution: a comprehensive survey , 2014, Machine Vision and Applications.

[35]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.