Visual-auditory integration for visual search: a behavioral study in barn owls

Barn owls are nocturnal predators that rely on both vision and hearing for survival. The optic tectum of barn owls, a midbrain structure involved in selective attention, has been used as a model for studying visual-auditory integration at the neuronal level. However, behavioral data on visual-auditory integration in barn owls are lacking. The goal of this study was to examine if the integration of visual and auditory signals contributes to the process of guiding attention toward salient stimuli. We attached miniature wireless video cameras on barn owls’ heads (OwlCam) to track their target of gaze. We first provide evidence that the area centralis (a retinal area with a maximal density of photoreceptors) is used as a functional fovea in barn owls. Thus, by mapping the projection of the area centralis on the OwlCam’s video frame, it is possible to extract the target of gaze. For the experiment, owls were positioned on a high perch and four food items were scattered in a large arena on the floor. In addition, a hidden loudspeaker was positioned in the arena. The positions of the food items and speaker were changed every session. Video sequences from the OwlCam were saved for offline analysis while the owls spontaneously scanned the room and the food items with abrupt gaze shifts (head saccades). From time to time during the experiment, a brief sound was emitted from the speaker. The fixation points immediately following the sounds were extracted and the distances between the gaze position and the nearest items and loudspeaker were measured. The head saccades were rarely toward the location of the sound source but to salient visual features in the room, such as the door knob or the food items. However, among the food items, the one closest to the loudspeaker had the highest probability of attracting a gaze shift. This result supports the notion that auditory signals are integrated with visual information for the selection of the next visual search target.

[1]  E. Knudsen,et al.  Signaling of the Strongest Stimulus in the Owl Optic Tectum , 2011, Journal of Neuroscience.

[2]  Katsumi Aoki,et al.  Recent development of flow visualization , 2004, J. Vis..

[3]  C. Spence,et al.  Cross-modal links in spatial attention. , 1998, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[4]  Y. Gutfreund,et al.  Saliency mapping in the optic tectum and its relationship to habituation , 2014, Front. Integr. Neurosci..

[5]  Shay Ohayon,et al.  Multiple manifestations of microstimulation in the optic tectum: eye movements, pupil dilations, and sensory priming. , 2010, Journal of neurophysiology.

[6]  G. Recanzone.  Rapidly induced auditory plasticity: the ventriloquism aftereffect. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[7]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[8]  T. Stanford,et al.  Evaluating the Operations Underlying Multisensory Integration in the Cat Superior Colliculus , 2005, The Journal of Neuroscience.

[9]  M. Goldberg,et al.  Neuronal Activity in the Lateral Intraparietal Area and Spatial Attention , 2003, Science.

[10]  E. A. Whitchurch,et al.  Combined auditory and visual stimuli facilitate head saccades in the barn owl (Tyto alba). , 2006, Journal of neurophysiology.

[11]  G. Recanzone.  Interactions of auditory and visual stimuli in space and time , 2009, Hearing Research.

[12]  A. J. King,et al.  Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus , 2004, Experimental Brain Research.

[13]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..

[14]  D. Ballard,et al.  Eye movements in natural behavior , 2005, Trends in Cognitive Sciences.

[15]  Matthew D. Hilchey,et al.  Driving forces in free visual search: An ethology , 2014, Attention, perception & psychophysics.

[16]  Shurong Wang,et al.  The nucleus isthmi and dual modulation of the receptive field of tectal neurons in non-mammals , 2003, Brain Research Reviews.

[17]  E I Knudsen,et al.  Neural maps of head movement vector and speed in the optic tectum of the barn owl. , 1990, Journal of neurophysiology.

[18]  Ehud Rivlin,et al.  On the barn owl’s visual pre-attack behavior: I. Structure of head movements and motion patterns , 2006, Journal of Comparative Physiology A.

[19]  Christoph Kayser,et al.  Functional Imaging Reveals Visual Modulation of Specific Fields in Auditory Cortex , 2007, The Journal of Neuroscience.

[20]  H. Wagner,et al.  Spatial attention modulates sound localization in barn owls. , 2001, Journal of neurophysiology.

[21]  S. Luck,et al.  On the role of selective attention in visual perception. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[22]  Mark T. Wallace,et al.  The Development of Cortical Multisensory Integration , 2006, The Journal of Neuroscience.

[23]  M. Alex Meredith,et al.  Neurons and behavior: the same rules of multisensory integration apply , 1988, Brain Research.

[24]  Y. Gutfreund,et al.  Stimulus-Specific Adaptation: Can It Be a Neural Correlate of Behavioral Habituation? , 2011, The Journal of Neuroscience.

[25]  E. Knudsen,et al.  Vision calibrates sound localization in developing barn owls , 1989, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[26]  R. Payne.  Acoustic location of prey by barn owls (Tyto alba). , 1971, The Journal of experimental biology.

[27]  N. Bolognini,et al.  Enhancement of visual perception by crossmodal visuo-auditory interaction , 2002, Experimental Brain Research.

[28]  E. Knudsen,et al.  Horizontal and vertical components of head movement are controlled by distinct neural circuits in the barn owl , 1990, Nature.

[29]  R. Berman,et al.  Attention and active vision , 2009, Vision Research.

[30]  G. Blasdel,et al.  Sound localization by the barn owl (Tyto alba) measured with the search coil technique , 1979, Journal of comparative physiology.

[31]  Peter König,et al.  Integrating audiovisual information for the control of overt attention. , 2007, Journal of vision.

[32]  Michael S Beauchamp,et al.  See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex , 2005, Current Opinion in Neurobiology.

[33]  Gerald E. Hough,et al.  Avian brains and a new understanding of vertebrate brain evolution , 2005, Nature Reviews Neuroscience.

[34]  Tom C T Yin,et al.  Bimodal Interactions in the Superior Colliculus of the Behaving Cat , 2002, The Journal of Neuroscience.

[35]  B. Stein,et al.  Spatial factors determine the activity of multisensory neurons in cat superior colliculus , 1986, Brain Research.

[36]  E. Knudsen.  Auditory and visual maps of space in the optic tectum of the owl , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[37]  Bruno B Averbeck,et al.  Integration of Auditory and Visual Communication Information in the Primate Ventrolateral Prefrontal Cortex , 2006, The Journal of Neuroscience.

[38]  J. Pettigrew,et al.  Quantitative analysis of the retinal ganglion cell layer and optic nerve of the barn owl Tyto alba. , 1989, Brain, behavior and evolution.

[39]  Jillian H. Fecteau,et al.  Salience, relevance, and firing: a priority map for target selection , 2006, Trends in Cognitive Sciences.

[40]  J. Pettigrew.  Binocular visual processing in the owl’s telencephalon , 1979, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[41]  S. Treue.  Visual attention: the where, what, how and why of saliency , 2003, Current Opinion in Neurobiology.

[42]  B. Stein,et al.  Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors , 1987, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[43]  Eric I. Knudsen,et al.  Global Inhibition and Stimulus Competition in the Owl Optic Tectum , 2010, The Journal of Neuroscience.

[44]  E I Knudsen.  Neural Derivation of Sound Source Location in the Barn Owl , 1987, Annals of the New York Academy of Sciences.

[45]  J. Wolfe,et al.  What attributes guide the deployment of visual attention and how do they do it? , 2004, Nature Reviews Neuroscience.

[46]  Ehud Rivlin,et al.  Through a barn owl’s eyes: interactions between scene content and visual attention , 2008, Biological Cybernetics.

[47]  T. Stanford,et al.  Development of multisensory integration from the perspective of the individual neuron , 2014, Nature Reviews Neuroscience.

[48]  Y. Gutfreund,et al.  Stimulus-Specific Adaptations in the Gaze Control System of the Barn Owl , 2008, The Journal of Neuroscience.

[49]  C. Spence,et al.  Attracting attention to the illusory location of a sound: reflexive crossmodal orienting and ventriloquism , 2000, Neuroreport.

[50]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[51]  Yoram Gutfreund,et al.  Multisensory enhancement in the optic tectum of the barn owl: spike count and spike timing. , 2009, Journal of neurophysiology.

[52]  T. Stanford,et al.  Multisensory integration: current issues from the perspective of the single neuron , 2008, Nature Reviews Neuroscience.

[53]  Timothy M. Woods,et al.  Visually Induced Plasticity of Auditory Spatial Perception in Macaques , 2004, Current Biology.

[54]  P Reinagel,et al.  Natural scene statistics at the centre of gaze. , 1999, Network.

[55]  M J Steinbach,et al.  Eye movements of the owl. , 1973, Vision research.

[56]  Miguel P Eckstein,et al.  Visual search: a retrospective. , 2011, Journal of vision.

[57]  M. Wallace,et al.  Representation and integration of multiple sensory inputs in primate superior colliculus. , 1996, Journal of neurophysiology.

[58]  Yoram Gutfreund,et al.  Stimulus-specific adaptation, habituation and change detection in the gaze control system , 2012, Biological Cybernetics.

[59]  Shreesh P Mysore,et al.  The role of a midbrain network in competitive stimulus selection , 2011, Current Opinion in Neurobiology.

[60]  M Konishi,et al.  Study of sound localization by owls and its relevance to humans. , 2000, Comparative biochemistry and physiology. Part A, Molecular & integrative physiology.

[61]  Eric I. Knudsen,et al.  Top-Down Control of Multimodal Sensitivity in the Barn Owl Optic Tectum , 2007, The Journal of Neuroscience.

[62]  Yoram Gutfreund,et al.  Auditory and Multisensory Responses in the Tectofugal Pathway of the Barn Owl , 2009, The Journal of Neuroscience.

[63]  Eric I. Knudsen,et al.  Gated Visual Input to the Central Auditory System , 2002, Science.

[64]  B. Stein,et al.  Interactions among converging sensory inputs in the superior colliculus. , 1983, Science.

[65]  Y. Gutfreund,et al.  Interactions between Stimulus-Specific Adaptation and Visual Auditory Integration in the Forebrain of the Barn Owl , 2010, The Journal of Neuroscience.

[66]  Eric I. Knudsen,et al.  A shared inhibitory circuit for both exogenous and endogenous control of stimulus selection , 2013, Nature Neuroscience.

[67]  Michelle C. Kwak,et al.  Visual and auditory cue integration for the generation of saccadic eye movements in monkeys and lever pressing in humans , 2012, The European journal of neuroscience.

[68]  Ohad Ben-Shahar,et al.  Overt attention toward oriented objects in free-viewing barn owls , 2011, Proceedings of the National Academy of Sciences of the United States of America.

[69]  D. Munoz,et al.  On the importance of the transient visual response in the superior colliculus , 2008, Current Opinion in Neurobiology.