Classical visual attention models only use visual features to predict where observers should look at. However, in daily life, visual information is never perceived without its corresponding audio signal. In a previous study, we found that sound modifies visual exploration by comparing the eye movements recorded when viewing videos with or without their original soundtrack. The aim of the presented research is to further understand how sound influences eye movements by controlling visual and audio contents of the videos, as well as the congruency between them. We describe an experiment with a novel approach in which observers watched videos belonging to four visual categories presenting different visual saliency distributions: landscapes, one moving object, several moving objects and faces. Videos were seen with their original soundtrack or with the soundtrack from another video belonging to the same visual category. Using different metrics to analyze the recorded eye movements, we found that sound has an influence only on videos containing faces and several moving objects. The original soundtrack decreases the variability between the eye positions of observers. Finally, we propose some cues to integrate sound information in classical visual attention models.
[1]
O. Meur,et al.
Predicting visual fixations on video based on low-level visual features
,
2007,
Vision Research.
[2]
Christof Koch,et al.
A Model of Saliency-Based Visual Attention for Rapid Scene Analysis
,
2009
.
[3]
Dorin Comaniciu,et al.
Mean Shift: A Robust Approach Toward Feature Space Analysis
,
2002,
IEEE Trans. Pattern Anal. Mach. Intell..
[4]
Antoine Coutrot,et al.
Influence of soundtrack on eye movements during video exploration
,
2012
.
[5]
Daniel T. Levin,et al.
A Window on Reality
,
2012
.
[6]
D. Senkowski,et al.
The multifaceted interplay between attention and multisensory integration
,
2010,
Trends in Cognitive Sciences.
[7]
Sidney S. Simon,et al.
Merging of the Senses
,
2008,
Front. Neurosci..
[8]
L. Itti,et al.
Quantifying center bias of observers in free viewing of dynamic natural scenes.
,
2009,
Journal of vision.
[9]
Nathalie Guyader,et al.
Modelling Spatio-Temporal Saliency to Predict Gaze Direction for Short Videos
,
2009,
International Journal of Computer Vision.