Computational and psycho-physiological investigations of musical emotions

In the first part of this paper we present a new computational modelling investigation of musical emotions based on connectionist spatiotemporal models. This connectionist model is able to consider both temporal dimensions (the dynamics of musical sequences) and spatial components (the parallel contribution of various music and psychoacoustic factors) to predict human participants emotional ratings of various music pieces. After training, the networks are able to successfully generalise the emotional ratings of music, including novel musical sequences. Detailed analyses of the networks parameters and dynamics allows us to identify the role of specific psychoacoustic variables (e.g. tempo, mean pitch, timbre etc.) in music emotional appraisal (in terms of Arousal and Valence). The second part of the paper describes an experiment that investigates the pattern of psycho-physiological emotional dynamics elicited by musical experience. It also focuses on the dynamics of Arousal and Valence (psychological dimensions), but it extends the measurements to physiological responses, namely Heart Rate and Skin Conductance Level. We report a preliminary analyses based on static (mean) and dynamic (second-by second measurements) relationships between sound and the affective dimensions considered. Results show strong relationships between linear combinations of the music dimensions Tempo and Loudness with psychological Arousal reports. For example, Arousal increases for higher levels of Tempo and/or Loudness. Data also show increased Heart Rate and Skin Conductance levels for sections with faster Tempo and higher Loudness. A negative relationship between Valence and Heart Rate was also found.

[1]  Aniruddh D. Patel,et al.  Temporal patterns of human cortical activity reflect tone sequence structure , 2000, Nature.

[2]  John F. Kolen,et al.  Resonance and the Perception of Musical Meter , 1994, Connect. Sci..

[3]  A. Lehmann,et al.  Tracking Performance Correlates of Changes in Perceived Intensity of Emotion During Different Interpretations of a Chopin Piano Prelude , 2001 .

[4]  W. Thompson,et al.  A Cross-Cultural Investigation of the Perception of Emotion in Music: Psychophysical and Cultural Cues , 1999 .

[5]  Garry Tamlyn,et al.  Music , 1993 .

[6]  K. Scherer,et al.  Handbook of affective sciences. , 2003 .

[7]  C. Habel,et al.  Language , 1931, NeuroImage.

[8]  R. Reisenzein.  The Schachter theory of emotion: two decades later. , 1983, Psychological bulletin.

[9]  Pierre Philippot,et al.  Respiratory feedback in the generation of emotion , 2002 .

[10]  R. Menezes,et al.  Entropy-Based Independence Test , 2006 .

[11]  R. Dolan,et al.  Emotion, Cognition, and Behavior , 2002, Science.

[12]  Macdonald Critchley,et al.  Music and the brain: Studies in the neurology of music Edited by Macdonald Critchley and R. A. Henson. Springfield, Illinois: Charles C Thomas, 1977. 459 pp , 1979, Brain and Language.

[13]  C. Krumhansl.  An exploratory study of musical emotions and psychophysiology. , 1997, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[14]  A. Damasio.  Descarte's error : emotion, reason, and the human brain , 1994 .

[15]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  J.M. Martins,et al.  Computational Musicology: An Artificial Life Approach , 2005, 2005 portuguese conference on artificial intelligence.

[17]  S H Strogatz,et al.  Coupled oscillators and biological synchronization. , 1993, Scientific American.

[18]  L. Cuddy,et al.  Expectancies generated by melodic intervals: Perceptual judgments of melodic continuity , 1995, Perception & psychophysics.

[19]  A. Damasio.  The Feeling of What Happens: Body and Emotion in the Making of Consciousness , 1999 .

[20]  J. Velásquez.  Modeling Emotion-Based Decision-Making , 1998 .

[21]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[22]  M. Washburn,et al.  Bodily Changes in Pain, Hunger, Fear, and Rage. , 1917 .

[23]  J. Sloboda,et al.  Psychological perspectives on music and emotion , 2001 .

[24]  Eduardo Coutinho,et al.  Evolving Emotional Behaviour for Expressive Performance of Music , 2005, IVA.

[25]  S. Vanderark,et al.  Cortisol, Biochemical, and Galvanic Skin Responses to Music Stimuli of Different Preference Values by College Students in Biology and Music , 1993, Perceptual and motor skills.

[26]  C. W. Hughes.  Emotion: Theory, Research and Experience , 1982 .

[27]  J. Sloboda.  The Musical Mind: The Cognitive Psychology of Music , 1987 .

[28]  J. Russell.  MEASURES OF EMOTION , 1989 .

[29]  Sandra Clara Gadanho,et al.  Robot Learning Driven by Emotions , 2001, Adapt. Behav..

[30]  A. Cangelosi,et al.  Towards a Model for Embodied Emotions , 2005, 2005 portuguese conference on artificial intelligence.

[31]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[32]  P. Ekman,et al.  Approaches To Emotion , 1985 .

[33]  C. Granger,et al.  USING THE MUTUAL INFORMATION COEFFICIENT TO IDENTIFY LAGS IN NONLINEAR MODELS , 1994 .

[34]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[35]  Domenico Parisi,et al.  Internal robotics , 2004, Connect. Sci..

[36]  P. Juslin,et al.  Emotional Expression in Music Performance: Between the Performer's Intention and the Listener's Experience , 1996 .

[37]  Nicola Dibben,et al.  The Role of Peripheral Feedback in Emotional Experience With Music , 2004 .

[38]  Bruce Bowe.  The Face of Emotion , 1985 .

[39]  R. Zatorre,et al.  Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[40]  Eduardo Reck Miranda,et al.  At the Crossroads of Evolutionary Computation and Music: Self-Programming Synthesizers, Swarm Orchestras and the Origins of Melody , 2004, Evolutionary Computation.

[41]  S. Schachter.  The Interaction of Cognitive and Physiological Determinants of Emotional State , 1964 .

[42]  Emery Schubert.  Measuring emotion continuously: Validity and reliability of the two-dimensional emotion-space , 1999 .

[43]  A. Cangelosi,et al.  The dynamics of music perception and emotional experience: a connectionist model , 2006 .

[44]  C. L. M.,et al.  Outlines of Psychology , 1891, Nature.

[45]  T. Dalgleish.  Basic Emotions , 2004 .

[46]  Carol L. Krumhansl,et al.  Melodic structure: Theoretical and empirical descriptions , 1991 .

[47]  Marketta Korhonen.  MODELING CONTINUOUS EMOTIONAL APPRAISALS OF MUSIC USING SYSTEM IDENTIFICATION , 2004 .

[48]  J. Russell.  A circumplex model of affect. , 1980 .

[49]  David Cope,et al.  Computers and Musical Style , 1993 .

[50]  Walter A. Siebel,et al.  Opinion TRENDS in Cognitive Sciences Vol.9 No.12 December 2005 Towards , 2022 .

[51]  Emery Schubert.  Modeling Perceived Emotion With Continuous Musical Features , 2004 .

[52]  K. Scherer,et al.  Cue utilization in emotion attribution from auditory stimuli , 1977 .

[53]  C. Krumhansl.  Music: A Link Between Cognition and Emotion , 2002 .

[54]  Phil Thomson,et al.  Atoms and errors: towards a history and aesthetics of microsound , 2004, Organised Sound.

[55]  A. Gabrielsson,et al.  The influence of musical structure on emotional expression. , 2001 .

[56]  Heinz Gregor Wieser.  Music and the brain. Lessons from brain diseases and some reflections on the "emotional" brain. , 2003, Annals of the New York Academy of Sciences.

[57]  Steven Jan,et al.  Replicating Sonorities: Towards a Memetics of Music , 2000 .

[58]  Eduardo Reck Miranda,et al.  A Connectionist Architecture for the Evolution of Rhythms , 2006, EvoWorkshops.

[59]  Densil Cabrera,et al.  ' PSYSOUND' : A COMPUTER PROGRAM FOR PSYCHOACOUSTICAL ANALYSIS , 1999 .

[60]  Simon Dixon,et al.  Automatic Extraction of Tempo and Beat From Expressive Performances , 2001 .

[61]  P. Juslin.  Communicating emotion in music performance: A review and a theoretical framework , 2001 .

[62]  N. Frijda,et al.  Emotions and respiratory patterns: review and critical analysis. , 1994, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[63]  K. Hevner.  Experimental studies of the elements of expression in music , 1936 .

[64]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[65]  E. Rolls,et al.  Memory systems in the brain. , 2000, Annual review of psychology.

[66]  M. Posner.  The Brain and Emotion , 1999, Nature Medicine.

[67]  Peter M. Todd,et al.  Frankensteinian methods for evolutionary music composition , 1999 .

[68]  Eduardo Reck Miranda,et al.  TOWARDS AN INTELLIGENT RHYTHMIC GENERATOR BASED ON GIVEN EXAMPLES : A MEMETIC APPROACH , 2005 .

[69]  Manfred Clynes,et al.  Sentics: The touch of emotions , 1977 .

[70]  L. Feldman.  Valence Focus and Arousal Focus: Individual Differences in the Structure of Affective Experience , 1995 .

[71]  C. Krumhansl.  A Perceptual Analysis of Mozart's Piano Sonata K. 282: Segmentation, Tension, and Musical Ideas , 1996 .

[72]  Paul Byers,et al.  Biological Rhythms as Information Channels in Interpersonal Communication Behavior , 1976 .

[73]  J. Sloboda.  Music Structure and Emotional Response: Some Empirical Findings , 1991 .

[74]  W. Dowling.  Emotion and Meaning in Music , 2008 .

[75]  A. Damasio.  Descartes' error: emotion, reason, and the human brain. avon books , 1994 .

[76]  Robert W. Kentridge.  Review: The Feeling of What Happens: Body, Emotion and the Making of Consciousness , 2000 .

[77]  Jônatas Manzolli.  An Evolutionary Approach Applied to Algorithmic Composition , 2004 .

[78]  Alan C. Evans,et al.  Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions , 1999, Nature Neuroscience.

[79]  Arnie Cox,et al.  The Mimetic Hypothesis and Embodied Musical Meaning , 2001 .

[80]  R. Lazarus.  Emotion and Adaptation , 1991 .

[81]  Jr. Horacio Fabrega.  The Feeling of What Happens: Body and Emotion in the Making of Consciousness , 2000 .

[82]  P. Todd,et al.  On Computational Models of the Evolution of Music: From the Origins of Musical Taste to the Emergence of Grammars , 2003 .

[83]  Cynthia Breazeal,et al.  Emotion and sociable humanoid robots , 2003, Int. J. Hum. Comput. Stud..

[84]  S. Tomkins.  Illuminating and Stimulating. (Book Reviews: Affect, Imagery, Consciousness. vol. 1, The Positive Affects) , 1963 .

[85]  J. Elman.  Distributed representations, simple recurrent networks, and grammatical structure , 1991, Machine Learning.

[86]  Sean A. Spence,et al.  Descartes' Error: Emotion, Reason and the Human Brain , 1995 .

[87]  R. Mcmurray,et al.  Effects of music on physiological and affective responses to graded treadmill exercise in trained and untrained runners. , 1995, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[88]  Stefan C. Kremer,et al.  Spatiotemporal Connectionist Networks: A Taxonomy and Review , 2001, Neural Computation.

[89]  N. Rickard.  Intense emotional responses to music: a test of the physiological arousal hypothesis , 2004 .

[90]  Eric D. Scheirer,et al.  Tempo and beat analysis of acoustic musical signals. , 1998, The Journal of the Acoustical Society of America.

[91]  Eduardo Reck Miranda,et al.  On the origins and evolution of music in virtual worlds , 2001 .

[92]  Iwanaga,et al.  Subjective and Physiological Responses to Music Stimuli Controlled Over Activity and Preference. , 1999, Journal of music therapy.

[93]  J. Panksepp.  Jaak Panksepp The Neuro-Evolutionary Cusp Between Emotions and Cognitions Implications for Understanding Consciousness and the Emergence of a Unified Mind Science , 2001 .

[94]  Eugene Narmour.  The Analysis and Cognition of Melodic Complexity: The Implication-Realization Model by Eugene Narmour": Notes , 1994 .

[95]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[96]  S. Khalfa,et al.  Event-related skin conductance responses to musical emotions in humans , 2002, Neuroscience Letters.

[97]  R. Parncutt.  Harmony: A Psychoacoustical Approach , 1989 .

[98]  Scott T. Grafton,et al.  Swinging in the brain: shared neural substrates for behaviors related to sequencing and music , 2003, Nature Neuroscience.

[99]  Phil Husbands,et al.  Musical Interaction with Artificial Life Forms: Sound Synthesis and Performance Mappings , 2003 .

[100]  Ah Chung Tsoi,et al.  Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical Inference , 2001, Machine Learning.

[101]  Eckart Altenmüller,et al.  EMuJoy: Software for continuous measurement of perceived emotions in music , 2007, Behavior research methods.

[102]  Dolores Canamero,et al.  A hormonal model of emotions for behavior control , 1997 .

[103]  Eduardo Miranda,et al.  Composing Music with Computers , 2001 .

[104]  J. Panksepp.  The emotional sources of "chills" induced by music. , 1995 .

[105]  Nuno R. Gracias,et al.  Gaia: An Artificial Life Environment for Ecological Systems Simulation-Artificial Life V: pp , 1997 .

[106]  J. Panksepp.  Affective Neuroscience: The Foundations of Human and Animal Emotions , 1998 .

[107]  J. Stainer,et al.  The Emotions , 1922, Nature.

[108]  Stephen N. Malloch,et al.  Timbre and technology : an analytical partnership : the development of an analytical technique and its application to music by Lutoslawski and Ligeti , 1997 .

[109]  K. Scherer.  Which Emotions Can be Induced by Music? What Are the Underlying Mechanisms? And How Can We Measure Them? , 2004 .

[110]  James W. Beauchamp,et al.  Machine Tongues XVI: Genetic Algorithms and Their Application to FM Matching Synthesis , 1993 .

[111]  S. Tomkins.  The positive affects , 1963 .

[112]  J. Sloboda,et al.  Music and emotion: Theory and research , 2001 .

[113]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[114]  J. Panksepp,et al.  Emotional sounds and the brain: the neuro-affective foundations of musical appreciation , 2002, Behavioural Processes.