Evidence for the integration of audiovisual emotional information at the perceptual level of processing

The present aim was to investigate how emotional expressions presented on an unattended channel affect the recognition of the attended emotional expressions. In Experiments 1 and 2, facial and vocal expressions were simultaneously presented as stimulus combinations. The emotions (happiness, anger, or emotional neutrality) expressed by the face and voice were either congruent or incongruent. Subjects were asked to attend either to the visual (Experiment 1) or auditory (Experiment 2) channel and recognise the emotional expression. The result showed that the ignored emotional expressions significantly affected the processing of attended signals as measured by recognition accuracy and response speed. In general, attended signals were recognised more accurately and faster in congruent than in incongruent combinations. In Experiment 3, possibility for a perceptual-level integration was eliminated by presenting the response-relevant and response-irrelevant signals separated in time. In this situation, emotional information presented on the nonattended channel ceased to affect the processing of emotional signals on the attended channel. The present results are interpreted to provide evidence for the view that facial and vocal emotional signals are integrated at the perceptual level of information processing and not at the later response-selection stages.

[1]  H. Schlosberg.  Three dimensions of emotion. , 1954, Psychological review.

[2]  P. J. Foley.  The foreperiod and simple reaction time. , 1959, Canadian journal of psychology.

[3]  Irwin Pollack,et al.  Communication of Verbal Modes of Expression , 1960 .

[4]  C. Osgood.  Dimensionality of the semantic space for communication via facial expressions. , 1966, Scandinavian journal of psychology.

[5]  A. Mehrabian,et al.  Inference of attitudes from nonverbal communication in two channels. , 1967, Journal of consulting psychology.

[6]  J. R. Simon,et al.  Auditory S-R compatibility: the effect of an irrelevant cue on information processing. , 1967, The Journal of applied psychology.

[7]  Herbert L. Pick,et al.  Selective attention to words and colors , 1972 .

[8]  K. Burns,et al.  Significance of vocal and visual channels in the decoding of emotional meaning. , 1973, The Journal of communication.

[9]  M. Posner,et al.  Attention and cognitive control. , 1975 .

[10]  P. Ekman.  Pictures of Facial Affect , 1976 .

[11]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[12]  B. Depaulo,et al.  Decoding discrepant nonverbal cues. , 1978 .

[13]  Barbara Dodd,et al.  Lip reading in infants: Attention to speech presented in- and out-of-synchrony , 1979, Cognitive Psychology.

[14]  J. Russell.  A circumplex model of affect. , 1980 .

[15]  P. Ekman,et al.  Relative importance of face, body, and speech in judgments of personality and affect. , 1980 .

[16]  R. Krauss,et al.  Verbal, vocal, and visible factors in judgments of another's affect. , 1981 .

[17]  U. Dimberg.  Facial reactions to facial expressions. , 1982, Psychophysiology.

[18]  A. Meltzoff,et al.  The bimodal perception of speech in infancy. , 1982, Science.

[19]  A. Walker-Andrews.  Intermodal perception of expressive behaviors: Relation of eye and voice? , 1986 .

[20]  K. Scherer,et al.  Cues and channels in emotion recognition. , 1986 .

[21]  K. Scherer.  Vocal affect expression: a review and a model for future research. , 1986, Psychological bulletin.

[22]  P. Ekman,et al.  DIFFERENCES Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion , 2004 .

[23]  J. Cutting,et al.  Minimodularity and the perception of layout. , 1988, Journal of experimental psychology. General.

[24]  A. Osman,et al.  Dimensional overlap: cognitive basis for stimulus-response compatibility--a model and taxonomy. , 1990, Psychological review.

[25]  D. Massaro,et al.  Perception of Synthesized Audible and Visible Speech , 1990 .

[26]  Colin M. Macleod.  Half a century of research on the Stroop effect: an integrative review. , 1991, Psychological bulletin.

[27]  T Hasbroucq,et al.  Stimulus-response compatibility and the Simon effect: toward a conceptual clarification. , 1991, Journal of experimental psychology. Human perception and performance.

[28]  A. Pick,et al.  Intermodal perception of happy and angry expressive behaviors by seven-month-old infants. , 1992, Child development.

[29]  R. Melara,et al.  Selective attention to Stroop dimensions: Effects of baseline discriminability, response mode, and practice , 1993, Memory & cognition.

[30]  D. Massaro,et al.  The paradigm and the fuzzy logical model of perception are alive and well. , 1993, Journal of experimental psychology. General.

[31]  H. Pashler.  Dual-task interference in simple tasks: data and theory. , 1994, Psychological bulletin.

[32]  S Kornblum,et al.  The way irrelevant dimensions are processed depends on what they overlap with: The case of Stroop- and Simon-like stimuli , 1994, Psychological research.

[33]  P. Barber,et al.  Stimulus congruence and the Simon effect , 1994, Psychological research.

[34]  J. Sergent,et al.  Segregated processing of facial identity and emotion in the human brain: A pet study , 1994 .

[35]  P. Lang.  The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[36]  L. Lundqvist,et al.  FACIAL EXPRESSIONS ARE CONTAGIOUS , 1995 .

[37]  Bernhard Hommel,et al.  Stimulus-response compatibility and the Simon effect: Toward an empirical clarification. , 1995 .

[38]  D. Massaro,et al.  Perceiving affect from the voice and the face , 1996, Psychonomic bulletin & review.

[39]  D. Massaro.  Perceiving talking faces: from speech perception to a behavioral principle , 1999 .

[40]  I. Linnankoski,et al.  Expression or emotional-motivational connotations with a one-word utterance. , 1997, The Journal of the Acoustical Society of America.

[41]  A. Nakamura,et al.  Vocal identification of speaker and emotion activates differerent brain regions , 1997, Neuroreport.

[42]  K. Edwards,et al.  The Face of Time: Temporal Cues in Facial Expressions of Emotion , 1998 .

[43]  J. Hietanen,et al.  Facial and emotional reactions to Duchenne and non-Duchenne smiles. , 1998, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[44]  Sönke Johannes,et al.  Brain potentials reveal the timing of face identity and expression judgments , 1998, Neuroscience Research.

[45]  J. Hietanen,et al.  Facial electromyographic responses to vocal affect expressions. , 1998, Psychophysiology.

[46]  W Gaebel,et al.  Neurophysiological correlates of the recognition of facial expressions of emotion as revealed by magnetoencephalography. , 1999, Brain research. Cognitive brain research.

[47]  Jyrki Tuomainen,et al.  The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses , 1999, Neuroscience Letters.

[48]  J. Vroomen,et al.  The perception of emotions by ear and by eye , 2000 .

[49]  A. Young,et al.  Configural information in facial expression perception. , 2000, Journal of experimental psychology. Human perception and performance.

[50]  B. Rossion,et al.  The time‐course of intermodal binding between seeing and hearing affective information , 2000, Neuroreport.

[51]  Jean Vroomen,et al.  Rejoinder - Bimodal emotion perception: integration across separate modalities, cross-modal perceptual grouping or perception of multimodal events? , 2000 .

[52]  L. Obler,et al.  Relationships among Facial, Prosodic, and Lexical Channels of Emotional Perceptual Processing , 2000 .

[53]  U. Dimberg,et al.  Facial reactions to happy and angry facial expressions: evidence for right hemisphere dominance. , 2000, Psychophysiology.

[54]  L D Rosenblum,et al.  Face and mouth inversion effects on visual and audiovisual speech perception. , 2000, Journal of experimental psychology. Human perception and performance.

[55]  D. Massaro,et al.  Fuzzy logical model of bimodal emotion perception: Comment on “The perception of emotions by ear and by eye” by de Gelder and Vroomen , 2000 .

[56]  H Bekkering,et al.  Motor activation from visible speech: evidence from stimulus response compatibility. , 2000, Journal of experimental psychology. Human perception and performance.

[57]  M. Erb,et al.  Are emotions contagious? Evoked emotions while viewing emotionally expressive faces: quality, quantity, time course and gender differences , 2001, Psychiatry Research.

[58]  Jon Driver,et al.  Is cross-modal integration of emotional expressions independent of attentional resources? , 2001, Cognitive, affective & behavioral neuroscience.

[59]  J. Ridley.  Studies of Interference in Serial Verbal Reactions , 2001 .

[60]  Mikko Sams,et al.  Does audiovisual speech perception use information about facial configuration? , 2001 .