The Consistency of Visual Search Models on High Dynamic Range and Tone Mapped Images

Tone mapping operators (TMO) are compression algorithms that compress the bit depth of high dynamic range (HDR) images in such a way that when the tone mapped version of the HDR image is shown on a low dynamic range screen, large portions of the image don’t appear over/under‐exposed. But the process of reducing the bit‐depth of an image often alters the appearance of the image due to the loss of information and the introduction of artifacts, changing the gaze patterns when people look at those tone mapped images. Saliency‐based TMOs aim to compress the bit‐depth of an HDR image while trying to keep the most salient locations in the HDR image salient after tone mapping. However, these models don’t ensure that the saliency model used is actually able to predict eye gaze in HDR environments accurately and assess how eye gaze is influenced by artifacts introduced into the image by the compression process. In this paper, we evaluate a suite of saliency models against 8 well‐known TMOs and the original HDR image to see how well each saliency model can predict the change in eye gaze when different TMOs are applied. By doing this, we can establish a firm basis on which to develop a saliency‐influenced tone mapping model to both compress HDR images and influence attention within the tone‐mapped image. If TMOs can selectively choose what to emphasize or de‐emphasize in a tone‐mapped image based on saliency results, then saliency‐based TMOs can be used to effectively direct attention even in complex environments.

[1]  Ali Borji,et al.  Computational models: Bottom-up and top-down aspects , 2015, ArXiv.

[2]  Karol Myszkowski,et al.  Adaptive Logarithmic Mapping For Displaying High Contrast Scenes , 2003, Comput. Graph. Forum.

[3]  Nanning Zheng,et al.  Learning to Detect a Salient Object , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  B. Tatler,et al.  Deriving an appropriate baseline for describing fixation behaviour , 2014, Vision Research.

[5]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Frédo Durand,et al.  Learning to predict where humans look , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  Alexei A. Efros,et al.  Fast bilateral filtering for the display of high-dynamic-range images , 2002 .

[8]  Esa Rahtu,et al.  Fast and Efficient Saliency Detection Using Sparse Sampling and Kernel Density Estimation , 2011, SCIA.

[9]  Frédo Durand,et al.  What Do Different Evaluation Metrics Tell Us About Saliency Models? , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Noel E. O'Connor,et al.  SalGAN: Visual Saliency Prediction with Generative Adversarial Networks , 2017, ArXiv.

[11]  Ali Borji,et al.  Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges , 2018, 1810.03716.

[12]  Michael Dorr,et al.  Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Panos Nasiopoulos,et al.  A learning-based visual saliency fusion model for High Dynamic Range video (LBVS-HDR) , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[14]  Rita Cucchiara,et al.  A deep multi-level network for saliency prediction , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[15]  Ali Borji,et al.  Computational models of attention , 2015, ArXiv.

[16]  Zhengguo Li,et al.  Visual-Salience-Based Tone Mapping for High Dynamic Range Images , 2014, IEEE Transactions on Industrial Electronics.

[17]  Frédo Durand,et al.  Learning Visual Importance for Graphic Designs and Data Visualizations , 2017, UIST.

[18]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[19]  Pietro Perona,et al.  Graph-Based Visual Saliency , 2006, NIPS.

[20]  Jessica K. Hodgins,et al.  Two methods for display of high contrast images , 1999, TOGS.

[21]  Aizaz Tirmizi,et al.  Pseudo-Multiple-Exposure-based Tone Fusion and Visual-Salience-based Tone Mapping for High Dynamic Range Images: A Review , 2015 .

[22]  Patrick Le Callet,et al.  Tone mapping based HDR compression: Does it affect visual experience? , 2014, Signal Process. Image Commun..

[23]  Wen-Chieh Lin,et al.  Attention-based high dynamic range imaging , 2011, The Visual Computer.

[24]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[25]  Jean-Philippe Tarel,et al.  Saliency Maps of High Dynamic Range Images , 2010, ECCV Workshops.

[26]  Michael J. Haass,et al.  Modeling Human Comprehension of Data Visualizations , 2017, HCI.

[27]  Jorma Laaksonen,et al.  Exploiting inter-image similarity and ensemble of extreme learners for fixation prediction using deep features , 2016, Neurocomputing.

[28]  Kenneth Chiu,et al.  Spatially Nonuniform Scaling Functions for High Contrast Images , 1993 .

[29]  Matthew H Tong,et al.  SUN: Top-down saliency using natural statistics , 2009, Visual cognition.

[30]  Tim K Marks,et al.  SUN: A Bayesian framework for saliency using natural statistics. , 2008, Journal of vision.

[31]  Panos Nasiopoulos,et al.  Human Visual System-Based Saliency Detection for High Dynamic Range Content , 2016, IEEE Transactions on Multimedia.

[32]  Andre Harrison,et al.  Enhanced visual perception through tone mapping , 2016, SPIE Defense + Security.

[33]  Dirk V. Arnold,et al.  Saliency-based parameter tuning for tone mapping , 2014, CVMP.

[34]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH.

[35]  Andre Harrison,et al.  An entropy based ideal observer model for visual saliency , 2012, 2012 46th Annual Conference on Information Sciences and Systems (CISS).

[36]  Ali Borji,et al.  Bottom-up Attention, Models of , 2018, Encyclopedia of Computational Neuroscience.

[37]  Giuseppe Valenzise,et al.  Evaluation of Feature Detection in HDR Based Imaging Under Changes in Illumination Conditions , 2015, 2015 IEEE International Symposium on Multimedia (ISM).

[38]  Frédo Durand,et al.  Where Should Saliency Models Look Next? , 2016, ECCV.

[39]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[40]  N. Aouf,et al.  Enhanced feature detection and matching under extreme illumination conditions with a HDR imaging sensor , 2013, 2012 IEEE 11th International Conference on Cybernetic Intelligent Systems (CIS).

[41]  Giuseppe Valenzise,et al.  An evaluation of HDR image matching under extreme illumination changes , 2016, 2016 Visual Communications and Image Processing (VCIP).

[42]  Rita Cucchiara,et al.  Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model , 2016, IEEE Transactions on Image Processing.

[43]  Manish Narwaria,et al.  Effect of tone mapping operators on visual attention deployment , 2012, Other Conferences.

[44]  Michael Ashikhmin,et al.  A reality check for tone-mapping operators , 2006, TAP.

[45]  Mark D. Fairchild,et al.  iCAM06: A refined image appearance model for HDR image rendering , 2007, J. Vis. Commun. Image Represent..

[46]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[47]  Aykut Erdem,et al.  Visual saliency estimation by nonlinearly integrating features using region covariances. , 2013, Journal of vision.

[48]  Ali Borji,et al.  State-of-the-Art in Visual Attention Modeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Stefan Mihalas,et al.  A model of proto-object based saliency , 2014, Vision Research.

[50]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[51]  Erik Reinhard,et al.  Photographic tone reproduction for digital images , 2002, ACM Trans. Graph..