360 Panorama Super-resolution using Deep Convolutional Networks

We propose deep convolutional neural network (CNN) based super-resolution for 360 (equirectangular) panorama images used by virtual reality (VR) display devices (e.g. VR glasses). Proposed super-resolution adopts the recent CNN architecture proposed in (Dong et al., 2016) and adapts it for equirectangular panorama images which have specific characteristics as compared to standard cameras (e.g. projection distortions). We demonstrate how adaptation can be performed by optimizing the trained network input size and fine-tuning the network parameters. In our experiments with 360 panorama images of rich natural content CNN based super-resolution achieves average PSNR improvement of 1.36 dB over the baseline (bicubic interpolation) and 1.56 dB by our equirectangular specific adaptation.

[1]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[3]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Munchurl Kim,et al.  Image super-resolution based on convolution neural networks using multi-channel input , 2016, 2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP).

[5]  Masashi Kameda,et al.  Single image super-resolution based on total variation regularization with Gaussian noise , 2016, 2016 Picture Coding Symposium (PCS).

[6]  Hong Chen,et al.  Matrix-value regression for single-image super-resolution , 2013, 2013 International Conference on Wavelet Analysis and Pattern Recognition.

[7]  Yuanying Qiu,et al.  A New Single Image Super-Resolution Method Based on the Infinite Mixture Model , 2017, IEEE Access.

[8]  Thomas S. Huang,et al.  Image super-resolution as sparse representation of raw image patches , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Olac Fuentes,et al.  Removing JPEG blocking artifacts using machine learning , 2014, 2014 Southwest Symposium on Image Analysis and Interpretation.

[10]  Batz Michel,et al.  Multi-image super-resolution using a locally adaptive denoising-based refinement , 2016 .

[11]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[12]  Li Guo,et al.  Image Super-Resolution with Deep Convolutional Neural Network , 2016, 2016 IEEE First International Conference on Data Science in Cyberspace (DSC).

[13]  Xiaohai He,et al.  Single image super-resolution based on deep learning and gradient transformation , 2016, 2016 IEEE 13th International Conference on Signal Processing (ICSP).

[14]  Horst Bischof,et al.  Fast and accurate image upscaling with super-resolution forests , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Fei Zhou,et al.  Interpolation-Based Image Super-Resolution Using Multisurface Fitting , 2012, IEEE Transactions on Image Processing.

[16]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Wan-Chi Siu,et al.  New motion compensation model via frequency classification for fast video super-resolution , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[18]  Wan-Chi Siu,et al.  Review of image interpolation and super-resolution , 2012, Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.