Audiovisual Interactions: The Impact of Visual Information on Music Perception and Memory

PREVIOUS RESEARCH HAS DEMONSTRATED THAT MUSICAL soundtracks can influence the emotional impact, interpretation, and remembering of visual information. The present research examines the reverse relationship and whether visual information influences the perception and memory of music. In Experiment 1, listeners were presented with affectively ambiguous tunes paired with visual displays varying in their affect (positive, negative) and format (video, montage), or a control condition (no visual information at all). After each, participants were asked to provide a set of perceptual ratings that evaluated different melody characteristics and qualities of the visual displays. Results showed that both the affect and format of visual information differentially influenced the way a melody was perceived. Experiment 2 extended these findings by revealing that the affect of visual displays distorted melody recognition in a mood congruent fashion. These results are discussed in terms of their theoretical implications for audiovisual processing.

[1]  Anna-Karin Weivert,et al.  Music and Emotion , 2022 .

[2]  John M. Geringer,et al.  Nonmusic Majors' Cognitive and Affective Responses to Performance and Programmatic Music Videos , 1997 .

[3]  Annabel J. Cohen,et al.  Associationism and musical soundtrack phenomena , 1993 .

[4]  E. Schellenberg,et al.  Good Pitch Memory Is Widespread , 2003, Psychological science.

[5]  K. Hevner.  Expression in music: a discussion of experimental studies and theories. , 1935 .

[6]  R. Welch.  Chapter 15 Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions , 1999 .

[7]  Cross-modality set effect on the perception of ambiguous pictures , 1976 .

[8]  David S. Levi.  The Structural Determinants of Melodic Expres Sive Properties , 1982 .

[9]  C. P. Heinlein.  The affective characters of the major and minor modes in music. , 1928 .

[10]  T. Bachmann,et al.  Cognitive contributions to the perception of spatial and temporal events (Advances in Psychology, Vol. 129) , 1999 .

[11]  P. Juslin.  Perceived Emotional Expression in Synthesized Performances of a Short Melody: Capturing the Listener's Judgment Policy , 1997 .

[12]  M. Boltz.  The cognitive processing of film and musical soundtracks , 2004, Memory & cognition.

[13]  L. Cosmides.  Invariances in the acoustic expression of emotion during speech. , 1983, Journal of experimental psychology. Human perception and performance.

[14]  J. W. Cassidy,et al.  Effects of Special Education Labels on Peers' and Adults' Evaluations of a Handicapped Youth Choir , 1991 .

[15]  A. Cohen.  Music as a source of emotion in film , 2001 .

[16]  C. Ryan,et al.  Attractiveness Bias in the Evaluation of Young Pianists' Performances , 2004 .

[17]  J. Montepare,et al.  The identification of emotions from gait information , 1987 .

[18]  M. Goldberg,et al.  Music, music videos, and wear out , 1993 .

[19]  M Boltz,et al.  Effects of background music on the remembering of filmed events , 1991, Memory & cognition.

[20]  A. Halpern,et al.  Memory for the absolute pitch of familiar songs , 1989, Memory & cognition.

[21]  John M. Geringer,et al.  Effects of Music with Video on Responses of Nonmusic Majors: An Exploratory Study , 1996 .

[22]  P. Lang,et al.  International Affective Picture System (IAPS): Instruction Manual and Affective Ratings (Tech. Rep. No. A-4) , 1999 .

[23]  Philip Glass,et al.  Koyaanisqatsi : life out of balance , 1983 .

[24]  M. Boltz.  Musical Soundtracks as a Schematic Influence on the Cognitive Processing of Filmed Events , 2001 .

[25]  K. Stevens,et al.  Emotions and speech: some acoustical correlates. , 1972, The Journal of the Acoustical Society of America.

[26]  K. Hevner,et al.  The affective value of pitch and tempo in music. , 1937 .

[27]  J. Sloboda,et al.  Psychological perspectives on music and emotion , 2001 .

[28]  A. Gabrielsson,et al.  The influence of musical structure on emotional expression. , 2001 .

[29]  Leonard B. Meyer.  Emotion and Meaning in Music , 1957 .

[30]  Michael Schutz,et al.  Hearing Gestures, Seeing Music: Vision Influences Perceived Tone Duration , 2007, Perception.

[31]  F. Russo,et al.  Facing the Music , 2007, Psychological science.

[32]  J. Wapnick,et al.  Effects of Performer Attractiveness, Stage Behavior, and Dress on Violin Performance Evaluation , 1998 .

[33]  Sallie Baxendale,et al.  Memories aren't made of this: amnesia at the movies , 2004, BMJ : British Medical Journal.

[34]  Kate Hevner,et al.  The affective character of the major and minor mode in music , 1935 .

[35]  A. Parrott.  Effects of Paintings and Music, Both Alone and in Combination, on Emotional Judgments , 1982 .

[36]  J. Wapnick,et al.  Effects of Selected Variables on Musicians’ Ratings of High-Level Piano Performances , 2004 .

[37]  James Lull,et al.  The Adolescent Audience for Music Videos and Why They Watch. , 1986 .

[38]  K. Scherer,et al.  Cue utilization in emotion attribution from auditory stimuli , 1977 .

[39]  Robert Gillespie.  Ratings of Violin and Viola Vibrato Performance in Audio-Only and Audiovisual Presentations , 1997 .

[40]  Annabel J. Cohen,et al.  Semantic and formal congruency in music and motion pictures : Effects on the interpretation of visual action , 1994 .

[41]  Antonio Camurri,et al.  Listeners' emotional engagement with performances of a Scriabin étude: an explorative case study , 2006 .

[42]  J. Russell.  A circumplex model of affect. , 1980 .

[43]  L Sloman,et al.  Gait patterns of depressed patients and normal subjects. , 1982, The American journal of psychiatry.

[44]  C. Krumhansl,et al.  Cross-modal interactions in the perception of musical performance , 2006, Cognition.

[45]  Fenguangzhai Song.  CD , 1992 .

[46]  W. Dowling.  Emotion and Meaning in Music , 2008 .

[47]  J. Wapnick,et al.  Effects of Physical Attractiveness on Evaluation of Vocal Performance , 1997 .

[48]  Claudia Bullerjahn,et al.  An empirical investigation of effects of film music using qualitative content analysis. , 1994 .

[49]  K. Scherer.  Acoustic Concomitants of Emotional Dimensions: Judging Affect from Synthesized Tone Sequences. , 1972 .