MC360IQA: The Multi-Channel CNN for Blind 360-Degree Image Quality Assessment

In this paper, we present a multi-channel convolution neural network (CNN) for blind 360-degree image quality assessment (MC360IQA). To be consistent with the visual content of 360-degree images seen in the VR device, our model adopts the viewport images as the input. Specifically, we project each 360-degree image into six viewport images to cover omnidirectional visual content. By rotating the longitude of the front view, we can project one omnidirectional image onto lots of different groups of viewport images, which is an efficient way to avoid overfitting. MC360IQA consists of two parts, multi-channel CNN and image quality regressor. Multi-channel CNN includes six parallel ResNet34 networks, which are used to extract the features of the corresponding six viewport images. Image quality regressor fuses the features and regresses them to final scores. The results show that our model achieves the best performance among the state-of-art full-reference (FR) and no-reference (NR) image quality assessment (IQA) models on the available 360-degree IQA database.

[1]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[2]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[3]  Henry Been-Lirn Duh,et al.  Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment , 2002, Proceedings IEEE Virtual Reality 2002.

[4]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[5]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[6]  Alan C. Bovik,et al.  No-Reference Image Quality Assessment in the Spatial Domain , 2012, IEEE Transactions on Image Processing.

[7]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[8]  Bernd Girod,et al.  A Framework to Evaluate Omnidirectional Video Coding Schemes , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality.

[9]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[10]  Lei Zhang,et al.  Blind Image Quality Assessment Using Joint Statistics of Gradient Magnitude and Laplacian Features , 2014, IEEE Transactions on Image Processing.

[11]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Touradj Ebrahimi,et al.  On the performance of objective metrics for omnidirectional visual content , 2017, 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX).

[14]  Lei Zhang,et al.  Learning without Human Scores for Blind Image Quality Assessment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[16]  Wei Sun,et al.  A Large-Scale Compressed 360-Degree Spherical Image Database: From Subjective Quality Evaluation to Objective Model Comparison , 2018, 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP).

[17]  Weisi Lin,et al.  CVIQD: Subjective quality evaluation of compressed virtual reality images , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[18]  Wenjun Zhang,et al.  Hybrid No-Reference Quality Metric for Singly and Multiply Distorted Images , 2014, IEEE Transactions on Broadcasting.

[19]  Xiongkuo Min,et al.  The prediction of head and eye movement for 360 degree images , 2018, Signal Process. Image Commun..

[20]  Vladyslav Zakharchenko,et al.  Quality metric for spherical panoramic video , 2016, Optical Engineering + Applications.