Multimodal Saliency Models for Videos

The modeling of visual attention has been the subject of numerous studies in many different research fields, from neurosciences to computer vision. This interdisciplinary interest led to the publication of a large number of computational models of attention, which emphasize salient regions in visual scenes. The current chapter provides an overview of the evolution of cognitive visual saliency models, from initial low-level models for static images to models for dynamic scenes using more complex features such as text or faces. We focus on saliency models combining multimodal features (especially audio and visual) into a single master saliency map. For instance, when computing the saliency of a video, these models jointly use auditory features extracted from the soundtrack and visual features from the visual frame. This is illustrated with the detailed description of an audiovisual saliency models for videos of conversations. This model includes a speaker diarization algorithm, which automatically modulates the saliency of conversation partners according to their speaking-or-not status. Finally, the chapter ends with some ideas to extend audiovisual saliency modeling to more general scenes featuring various contents.