A Multimodal View into Music's Effect on Human Neural, Physiological, and Emotional Experience

Music has a powerful influence on human experience. In this paper, we investigate how music affects brain activity, physiological response, and human-reported behavior. Using auditory features related to dynamics, timbre, harmony, rhythm, and register, we predicted brain activity in the form of phase synchronizations in bilateral Heschl's gyri and superior temporal gyri; physiological response in the form of galvanic skin response and heart activity; and emotional experience in the form of continuous, subjective descriptions reported by music listeners. We found that using multivariate time series models with attention mechanisms are effective in predicting emotional ratings, while vector-autoregressive models are effective in predicting involuntary human responses. Musical features related to dynamics, register, rhythm, and harmony were found to be particularly helpful in predicting these human reactions. This work adds to our understanding of how music affects multimodal human experience and has applications in affective computing, music emotion recognition, neuroscience, and music information retrieval.

[1]  Mikko Sams,et al.  Large-scale brain networks emerge from dynamic processing of musical timbre, key and rhythm , 2012, NeuroImage.

[2]  Andreas F. Ehmann,et al.  Lyric Text Mining in Music Mood Classification , 2009, ISMIR.

[3]  D. Conforti,et al.  Heart Rate Response to Music An Artificial Intelligence Study on Healthy and Traumatic Brain-Injured Subjects , 2008 .

[4]  Kivanç Tatar,et al.  Ranking-Based Emotion Recognition for Experimental Music , 2017, ISMIR.

[5]  Vinoo Alluri,et al.  Capturing the musical brain with Lasso: Dynamic decoding of musical features from fMRI data , 2014, NeuroImage.

[6]  Tanaya Guha,et al.  Affective Feature Design and Predicting Continuous Affective Dimensions from Music , 2014, MediaEval.

[7]  Kirk Martinez,et al.  Music genre classification using polyphonic timbre models , 2014, 2014 19th International Conference on Digital Signal Processing.

[8]  F. Shaffer,et al.  An Overview of Heart Rate Variability Metrics and Norms , 2017, Front. Public Health.

[9]  F. Martin,et al.  Musical chords and emotion: Major and minor triads are processed for emotion , 2015, Cognitive, affective & behavioral neuroscience.

[10]  Petri Toiviainen,et al.  A Matlab Toolbox for Music Information Retrieval , 2007, GfKl.

[11]  Grigorios Tsoumakas,et al.  Multi-Label Classification of Music into Emotions , 2008, ISMIR.

[12]  Hung-yi Lee,et al.  Temporal pattern attention for multivariate time series forecasting , 2018, Machine Learning.

[13]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986 .

[14]  Daniel J. Levitin,et al.  The rewards of music listening: Response and physiological connectivity of the mesolimbic system , 2005, NeuroImage.

[15]  Wolfgang Nejdl,et al.  Music Mood and Theme Classification - a Hybrid Approach , 2009, ISMIR.

[16]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[17]  S. Koelsch.  Investigating Emotion with Music , 2005, Annals of the New York Academy of Sciences.

[18]  Istvan Molnar-Szakacs,et al.  Beyond superior temporal cortex: intersubject correlations in narrative speech comprehension. , 2008, Cerebral cortex.

[19]  Kaustubh Supekar,et al.  Sparse logistic regression for whole-brain classification of fMRI data , 2010, NeuroImage.

[20]  Ching-Hua Chuan,et al.  Modeling Temporal Tonal Relations in Polyphonic Music Through Deep Networks With a Novel Image-Based Representation , 2018, AAAI.

[21]  Sheng Chen,et al.  NARX-Based Nonlinear System Identification Using Orthogonal Least Squares Basis Hunting , 2008, IEEE Transactions on Control Systems Technology.

[22]  Guokun Lai,et al.  Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks , 2017, SIGIR.

[23]  Abdelhamid Bouchachia,et al.  Ensemble Learning for Time Series Prediction , 2008 .

[24]  Jeffrey J. Scott,et al.  MUSIC EMOTION RECOGNITION: A STATE OF THE ART REVIEW , 2010 .

[25]  D. Ellis,et al.  Effects of music on respiration- and heart-rate. , 1952, The American journal of psychology.

[26]  José Fornari,et al.  Multi-Feature Modeling of Pulse Clarity: Design, Validation and Optimization , 2008, ISMIR.

[27]  Emmanuel Dellandréa,et al.  What is the best segment duration for music mood analysis ? , 2008, 2008 International Workshop on Content-Based Multimedia Indexing.

[28]  Mikko Sams,et al.  Functional Magnetic Resonance Imaging Phase Synchronization as a Measure of Dynamic Functional Connectivity , 2012, Brain Connect..

[29]  Carl E. Rasmussen,et al.  Variational Gaussian Process State-Space Models , 2014, NIPS.

[30]  Björn W. Schuller,et al.  Determination of Nonprototypical Valence and Arousal in Popular Music: Features and Performances , 2010, EURASIP J. Audio Speech Music. Process..

[31]  C. Stevens,et al.  Sweet Anticipation: Music and the Psychology of Expectation, by David Huron . Cambridge, Massachusetts: MIT Press, 2006 , 2007 .

[32]  Philippe Pasquier,et al.  Automatic Soundscape Affect Recognition Using A Dimensional Approach , 2016 .

[33]  Pascal Belin,et al.  Stimulus Complexity and Categorical Effects in Human Auditory Cortex: An Activation Likelihood Estimation Meta-Analysis , 2011, Front. Psychology.

[34]  Robert F. Potter,et al.  Effects of Music on Physiological Arousal: Explorations into Tempo and Genre , 2007 .

[35]  Andrea Stocco,et al.  Coordinate-Based Meta-Analysis of fMRI Studies with R , 2014, R J..

[36]  Y. Song,et al.  A Survey of Music Recommendation Systems and Future Perspectives , 2012 .

[37]  Lie Lu,et al.  Automatic mood detection and tracking of music audio signals , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[38]  Shrikanth Narayanan,et al.  Learning Shared Vector Representations of Lyrics and Chords in Music , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[39]  Gert R. G. Lanckriet,et al.  Semantic Annotation and Retrieval of Music and Sound Effects , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[40]  F. Riganello,et al.  Heart Rate Variability, Emotions, and Music , 2010 .

[41]  Stefan Koelsch,et al.  Adults and children processing music: An fMRI study , 2005, NeuroImage.

[42]  Talma Hendler,et al.  Common modulation of limbic network activation underlies musical emotions as they unfold , 2016, NeuroImage.

[43]  M. Trumper.  Bodily Changes in Pain, Hunger, Fear and Rage: An Account of Recent Researches into the Function of Emotional Excitement , 1930, The Psychological Clinic.

[44]  M. Tervaniemi,et al.  A Functional MRI Study of Happy and Sad Emotions in Music with and without Lyrics , 2011, Front. Psychology.

[45]  Seungmin Rho,et al.  Music emotion classification and context-based music recommendation , 2010, Multimedia Tools and Applications.

[46]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition from Brain Signals Using Hybrid Adaptive Filtering and Higher Order Crossings Analysis , 2010, IEEE Transactions on Affective Computing.

[47]  S. Khalfa,et al.  Event-related skin conductance responses to musical emotions in humans , 2002, Neuroscience Letters.

[48]  Shrikanth S. Narayanan,et al.  Sparse Representation of Electrodermal Activity With Knowledge-Driven Dictionaries , 2015, IEEE Transactions on Biomedical Engineering.

[49]  Thomas M. Cover,et al.  Elements of information theory (2. ed.) , 2006 .

[50]  S. McAdams,et al.  The Present, Past, and Future of Timbre Research , 2019, Timbre: Acoustics, Perception, and Cognition.

[51]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition From EEG Using Higher Order Crossings , 2010, IEEE Transactions on Information Technology in Biomedicine.

[52]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[53]  B. Hansen.  TIME SERIES ANALYSIS James D. Hamilton Princeton University Press, 1994 , 1995, Econometric Theory.

[54]  Marc Thomas Philipp Adam,et al.  Measuring Emotional Arousal for Online Applications: Evaluation of Ultra-short Term Heart Rate Variability Measures , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.