Putting the pieces together: Revealing face–voice integration through the facial overshadowing effect

ABSTRACT Two experiments are reported which explore the limits of the facial overshadowing effect on voice recognition. In Experiment 1, the facial overshadowing effect was replicated. Voice recognition was significantly impaired when the voice had been studied in the presence of the face rather than when studied alone. However, this effect was removed when the accompanying face was inverted. In Experiment 2, the facial overshadowing effect was replicated using more realistic moving faces rather than static faces through presentation of a video sequence at study. Surprisingly, the magnitude of the facial overshadowing effect was not influenced by whether faces were static or dynamic. Moreover, the effect remained even under temporal asynchrony. Together, these results show the facial overshadowing effect to be a robust phenomenon. However, it appears to depend critically on one factor: the facial nature of the distraction. The results are discussed in terms of the integration of faces and voices in a recognition context, and in terms of a holistic view of face–voice processing in a more general person perception framework. This offers an explanation of current findings, and serves to guide predictions for future work.

[1]  S. Campanella,et al.  Cross-modal interactions between human faces and voices involved in person recognition , 2011, Cortex.

[2]  F. Gosselin,et al.  Audio-visual integration of emotion expression , 2008, Brain Research.

[3]  S. M. Sheffert,et al.  Audiovisual speech facilitates voice learning , 2004, Perception & psychophysics.

[4]  K. Lander,et al.  Why are moving faces easier to recognize? , 2005, Visual Cognition.

[5]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[6]  A. Young,et al.  Understanding person perception. , 2011, British journal of psychology.

[7]  Galit Yovel,et al.  Recognizing People in Motion , 2016, Trends in Cognitive Sciences.

[8]  K. Scherer Vocal affect expression: a review and a model for future research. , 1986, Psychological bulletin.

[9]  R. Yin Looking at Upside-down Faces , 1969 .

[10]  P. Belin,et al.  Understanding voice perception. , 2011, British journal of psychology.

[11]  J. Wilding,et al.  Earwitness testimony: effects of exposure and attention on the face overshadowing effect. , 2001, British journal of psychology.

[12]  P. Belin,et al.  Thinking the voice: neural correlates of voice perception , 2004, Trends in Cognitive Sciences.

[13]  P. Laukka,et al.  A dimensional approach to vocal expression of emotion , 2005 .

[14]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[15]  Alfred Anwander,et al.  Direct Structural Connections between Voice- and Face-Recognition Areas , 2011, The Journal of Neuroscience.

[16]  G. Gainotti Laterality effects in normal subjects' recognition of familiar faces, voices and names. Perceptual and representational components , 2013, Neuropsychologia.

[17]  Gordon E. Legge,et al.  Learning unfamiliar voices , 1984 .

[18]  Earwitness testimony 2. Voices, faces and context , 1997 .

[19]  A. D. Yarmey Verbal, visual, and voice identification of a rape suspect under different levels of illumination. , 1986, The Journal of applied psychology.

[20]  Andreas Kleinschmidt,et al.  Interaction of Face and Voice Areas during Speaker Recognition , 2005, Journal of Cognitive Neuroscience.

[21]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[22]  A. Yarmey EARWITNESS SPEAKER IDENTIFICATION , 1995 .

[23]  S. Schweinberger,et al.  Benefits for Voice Learning Caused by Concurrent Faces Develop over Time , 2015, PloS one.

[24]  S. Schweinberger,et al.  The role of audiovisual asynchrony in person recognition , 2010, Quarterly journal of experimental psychology.

[25]  Michael Erb,et al.  Cerebral integration of verbal and nonverbal emotional cues: Impact of individual nonverbal dominance , 2012, NeuroImage.

[26]  Allyssa McCabe,et al.  When Eyewitnesses Are Also Earwitnesses: Effects on Visual and Voice Identifications , 1993 .

[27]  Keeley Moore,et al.  Earwitness Memory: Effects of Facial Concealment on the Face Overshadowing Effect , 2011 .

[28]  Greg J. Neil,et al.  When the face fits: Recognition of celebrities from matching and mismatching faces and voices , 2014, Memory.

[29]  Interference in eyewitness and earwitness recognition , 2011 .

[30]  S. McKelvie,et al.  Effect of face context on recognition memory for voices. , 1996, The Journal of general psychology.

[31]  S. Campanella,et al.  Integrating face and voice in person perception , 2007, Trends in Cognitive Sciences.

[32]  S. Schweinberger,et al.  Hearing Facial Identities , 2007, Quarterly journal of experimental psychology.

[33]  Greg J. Neil,et al.  Hearing Faces and Seeing Voices: The Integration and Interaction of Face and Voice Processing , 2014 .