The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses

Judgement of the emotional tone of a spoken utterance is influenced by a simultaneously presented face expression. The time course of this integration was investigated by measuring the mismatch negativity (MMN). In one condition, the standard stimulus was an angry voice fragment combined with a (congruous) angry face expression. In the deviant pair, the voice expression was kept the same and only the face expression changed to an (incongruous) sad face. The pairs with a deviant visual item evoked a negative electric brain response showing the characteristics of the MMN, which is usually evoked only by auditory deviations. Similar results were obtained by employing incongruous standard and congruous deviant pairs. These findings provide compelling evidence of an early integration of face with voice information in the processing of affect.

[1]  Paul Bertelson,et al.  The ventriloquist effect does not depend on the direction of visual attention , 2000 .

[2]  D. Massaro,et al.  Perceiving affect from the voice and the face , 1996, Psychonomic bulletin & review.

[3]  J. Hietanen,et al.  Modulation of human auditory information processing by emotional visual stimuli. , 1998, Brain research. Cognitive brain research.

[4]  S. Hillyard,et al.  The effects of channel-selective attention on the mismatch negativity wave elicited by deviant tones. , 1991, Psychophysiology.

[5]  D. Jeffreys.  Evoked Potential Studies of Face and Object Processing , 1996 .

[6]  R. Ilmoniemi,et al.  Language-specific phoneme representations revealed by electric and magnetic brain responses , 1997, Nature.

[7]  Anders M. Dale,et al.  Electrical and magnetic readings of mental functions , 1997 .

[8]  J. Vroomen,et al.  Impaired perception of face and emotion in Huntington's disease , 1998 .

[9]  E. Donchin,et al.  COGNITIVE PSYCHOPHYSIOLOGY: THE ENDOGENOUS COMPONENTS OF THE ERP , 1978 .

[10]  D. Perrett,et al.  A differential neural response in the human amygdala to fearful and happy facial expressions , 1996, Nature.

[11]  Word recognition time and event related potentials during emotional word processing , 1990 .

[12]  P. Bertelson,et al.  The ventriloquist effect does not depend on the direction of deliberate visual attention , 2000, Perception & psychophysics.

[13]  E. Rolls,et al.  Selectivity between faces in the responses of a population of neurons in the cortex in the superior temporal sulcus of the monkey , 1985, Brain Research.

[14]  K Alho,et al.  Mismatch negativity to auditory stimulus change recorded directly from the human temporal cortex. , 1995, Psychophysiology.

[15]  J. Vroomen,et al.  The perception of emotions by ear and by eye , 2000 .

[16]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[17]  B. Stein,et al.  The Merging of the Senses , 1993 .

[18]  R. Näätänen.  Attention and brain function , 1992 .

[19]  G. Ojemann,et al.  Organization of visuospatial functions in human cortex. Evidence from electrical stimulation. , 1982, Brain : a journal of neurology.

[20]  M. M. van den Berg-Lenssen,et al.  Correction of ocular artifacts in EEGs using an autoregressive model to describe the EEG; a pilot study. , 1989, Electroencephalography and clinical neurophysiology.

[21]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.