V-BMS360: A Video Extention to the BMS360 Image Saliency Model

In this paper it is studied how existing image saliency models for head motion prediction designed for omnidirectional images can be applied to videos. A new model called V-BMS360 which extends the BMS360 model is presented. Due to the specific properties of omnidirectional videos, new key features had to be introduced: the first one is the introduction of a temporal prior that accounts for the delay needed by the users to explore the content during the first seconds of the videos. The second key novel idea is the consideration of camera motion and its consequences on the exploration of the visual scenes. It was translated into two key aspects new features: the “motion surroundness” and the “motion source”, and an informed pooling allowing to provide different strengths to different motion-based features based on a camera-motion analysis. All of these contribute to the performance of the new saliency model called V-BMS360.

[1]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[2]  E. Freedman.  Coordination of the eyes and head during visual orienting , 2008, Experimental Brain Research.

[3]  Antoine Coutrot,et al.  A dataset of head and eye movements for 360° videos , 2018, MMSys.

[4]  Mikhail Startsev,et al.  360-aware Saliency Estimation with Conventional Image Saliency Predictors , 2018, Signal Process. Image Commun..

[5]  Alexander Raake,et al.  GBVS360, BMS360, ProSal: Extending existing saliency prediction models from 2D to omnidirectional images , 2018, Signal Process. Image Commun..

[6]  Ming-Yu Liu,et al.  Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Videos , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ali Borji,et al.  Revisiting Video Saliency: A Large-Scale Benchmark and a New Model , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Zhenzhong Chen,et al.  A saliency prediction model on 360 degree images using color dictionary based sparse representation , 2018, Signal Process. Image Commun..

[9]  Cagri Ozcinar,et al.  Look around you: Saliency maps for omnidirectional images in VR applications , 2017, 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX).

[10]  Gordon Wetzstein,et al.  Saliency in VR: How Do People Explore Virtual Environments? , 2016, IEEE Transactions on Visualization and Computer Graphics.

[11]  Zulin Wang,et al.  Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTM , 2017, ECCV.

[12]  Patrick Le Callet,et al.  Toolbox and dataset for the development of saliency and scanpath models for omnidirectional/360° still images , 2018, Signal Process. Image Commun..

[13]  Helge J. Ritter,et al.  Visual search in the (un)real world: how head-mounted displays affect eye movements, head movements and target detection , 2010, ETRA '10.

[14]  Rafael Monroy,et al.  SalNet360: Saliency Maps for omni-directional images with CNN , 2017, Signal Process. Image Commun..