Towards User-Aware Music Information Retrieval: Emotional and Color Perception of Music

This chapter presents our findings on emotional and color perception of music. It emphasizes the importance of user-aware music information retrieval (MIR) and the advantages that research on emotional processing and interaction between multiple modalities brings to the understanding of music and its users. Analyses of results show that correlations between emotions, colors and music are largely determined by context. There are differences between emotion-color associations and valence-arousal ratings in non-music and music contexts, with the effects of genre preferences evident for the latter. Participants were able to differentiate between perceived and induced musical emotions. Results also show how associations between individual musical emotions affect their valence-arousal ratings. We believe these findings contribute to the development of user-aware MIR systems and open further possibilities for innovative applications in MIR and affective computing in general.

[1]  Charles Spence,et al.  ‘When Birds of a Feather Flock Together’: Synesthetic Correspondences Modulate Audiovisual Integration in Non-Synesthetes , 2009, PloS one.

[2]  Paul Lamere,et al.  Using 3D Visualizations to Explore and Discover Music , 2007, ISMIR.

[3]  N. Dibben,et al.  Emotion and music: A view from the cultural psychology of music , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[4]  Jeffrey J. Scott,et al.  MUSIC EMOTION RECOGNITION: A STATE OF THE ART REVIEW , 2010 .

[5]  Ilana B. Witten,et al.  Why Seeing Is Believing: Merging Auditory and Visual Worlds , 2005, Neuron.

[6]  Fabio Vignoli,et al.  Visual Playlist Generation on the Artist Map , 2005, ISMIR.

[7]  M. Naumer,et al.  Semantics and the multisensory brain: How meaning modulates processes of audio-visual integration , 2008, Brain Research.

[8]  F. Reis,et al.  Multi-Modal Emotion Music Recognition (MER): A New Dataset, Methodology and Comparative Analysis , 2011 .

[9]  K. Scherer Which Emotions Can be Induced by Music? What Are the Underlying Mechanisms? And How Can We Measure Them? , 2004 .

[10]  L. Fabrigar,et al.  Reexamining the Circumplex Model of Affect , 2000 .

[11]  Meinard Müller,et al.  Automated Methods for Analyzing Music Recordings in Sonata Form , 2013, ISMIR.

[12]  C. Spence Audiovisual multisensory integration , 2007 .

[13]  P. Laukka,et al.  Expression, Perception, and Induction of Musical Emotions: A Review and a Questionnaire Study of Everyday Listening , 2004 .

[14]  Yi-Hsuan Yang,et al.  1000 songs for emotional analysis of music , 2013, CrowdMM '13.

[15]  A. Gabrielsson Emotion perceived and emotion felt: Same or different? , 2001 .

[16]  T. Eerola Are the Emotions Expressed in Music Genre-specific? An Audio-based Evaluation of Datasets Spanning Classical, Film, Pop and Mixed Genres , 2011 .

[17]  Hsin-Min Wang,et al.  A histogram density modeling approach to music emotion recognition , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  Gerhard Widmer,et al.  Exploring Music Collections by Browsing Different Views , 2004, Computer Music Journal.

[19]  Anne Treisman,et al.  Natural cross-modal mappings between visual and auditory features. , 2011, Journal of vision.

[20]  J. Marozeau,et al.  Multidimensional scaling of emotional responses to music: The effect of musical expertise and of the duration of the excerpts , 2005 .

[21]  György Fazekas,et al.  Multidisciplinary Perspectives on Music Emotion Recognition: Implications for Content and Context-Based Models , 2012 .

[22]  D. Västfjäll,et al.  Emotional responses to music: the need to consider underlying mechanisms. , 2008, The Behavioral and brain sciences.

[23]  Meinard Müller,et al.  Converting Path Structures Into Block Structures Using Eigenvalue Decompositions of Self-Similarity Matrices , 2013, ISMIR.

[24]  Youngmoo E. Kim,et al.  Exploring automatic music annotation with "acoustically-objective" tags , 2010, MIR '10.

[25]  Tuomas Eerola,et al.  Semantic Computing of Moods Based on Tags in Social Media of Music , 2013, IEEE Transactions on Knowledge and Data Engineering.

[26]  Tuomas Eerola,et al.  A Review of Music and Emotion Studies: Approaches, Emotion Models, and Stimuli , 2013 .

[27]  J. Russell A circumplex model of affect. , 1980 .

[28]  Tuomas Eerola,et al.  Modeling Listeners' Emotional Response to Music , 2012, Top. Cogn. Sci..

[29]  Yi-Hsuan Yang,et al.  Exploring the relationship between categorical and dimensional emotion semantics of music , 2012, MIRUM '12.

[30]  Gert R. G. Lanckriet,et al.  Semantic Annotation and Retrieval of Music and Sound Effects , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[31]  Björn Schuller,et al.  ‘Mister D.J., Cheer Me Up!’: Musical and Textual Features for Automatic Mood Classification , 2010 .

[32]  G. Calvert Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[33]  E. Pampalk Islands of Music Analysis, Organization, and Visualization of Music Archives , 2002 .

[34]  Markus Schedl,et al.  Putting the User in the Center of Music Information Retrieval , 2012, ISMIR.

[35]  Yi-Hsuan Yang,et al.  Machine Recognition of Music Emotion: A Review , 2012, TIST.

[36]  Remco C. Veltkamp,et al.  Designing Games with a Purpose for Data Collection in Music Research. Emotify and Hooked: Two Case Studies , 2013, GALA.

[37]  Fabien Gouyon,et al.  MIRrors: Music Information Research reflects on its future , 2013, Journal of Intelligent Information Systems.

[38]  Petri Toiviainen,et al.  Prediction of Multidimensional Emotional Ratings in Music from Audio Using Multivariate Regression Models , 2009, ISMIR.

[39]  Stephen Lakatos,et al.  Cross-modal interactions in auditory and visual discrimination. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[40]  Gregor Strle,et al.  Capturing the mood: Evaluation of the moodstripe and moodgraph interfaces , 2014, 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[41]  Daniel Müllensiefen,et al.  The Musicality of Non-Musicians: An Index for Assessing Musical Sophistication in the General Population , 2014, PloS one.

[42]  Tuomas Eerola,et al.  Measuring music-induced emotion , 2011 .

[43]  T. Eerola,et al.  A comparison of the discrete and dimensional models of emotion in music , 2011 .

[44]  Tatsuki Imai,et al.  Chord-Cube: Music Visualization and Navigation System with an Emotion-Aware Metric Space for Temporal Chord Progression , 2014 .

[45]  L. Ou,et al.  A study of colour emotion and colour preference. Part I: Colour emotions for single colours , 2004 .

[46]  L. Ou,et al.  A study of colour emotion and colour preference. Part III: Colour preference modeling , 2004 .

[47]  Catherine J. Stevens,et al.  Music Perception and Cognition: A Review of Recent Cross-Cultural Research , 2012, Top. Cogn. Sci..

[48]  Remco C. Veltkamp,et al.  A Survey of Music Information Retrieval Systems , 2005, ISMIR.

[49]  K. Scherer,et al.  EMOTIONAL EFFECTS OF MUSIC: PRODUCTION RULES , 2001 .

[50]  Xavier Serra,et al.  Roadmap for Music Information ReSearch , 2013 .

[51]  W. Fitch,et al.  Beyond Intensity: Spectral Features Effectively Predict Music-Induced Subjective Arousal , 2014, Quarterly journal of experimental psychology.

[52]  Didier Grandjean,et al.  Towards a Dynamic Approach to the Study of Emotions Expressed by Music , 2011, INTETAIN.

[53]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[54]  D. Levitin,et al.  Current Advances in the Cognitive Neuroscience of Music , 2009, Annals of the New York Academy of Sciences.

[55]  Tuomas Eerola,et al.  Modelling emotional effects of music : key areas of improvement. , 2013 .

[56]  F. Gosselin,et al.  Audio-visual integration of emotion expression , 2008, Brain Research.

[57]  Stephen E. Palmer,et al.  The Color of Musical Sounds: Color Associates of Harmony and Timbre in Non-Synesthetes , 2012 .

[58]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[59]  Sally Jo Cunningham,et al.  The Impact (or Non-impact) of User Studies in Music Information Retrieval , 2012, ISMIR.

[60]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[61]  Masataka Goto,et al.  Music Thumbnailer: Visualizing Musical Pieces in Thumbnail Images Based on Acoustic Features , 2008, ISMIR.

[62]  Wolfgang Sanz,et al.  Laser-optical investigation of stator-rotor interaction in a transonic turbine , 2006, J. Vis..

[63]  R. Paiva,et al.  Multi-Modal Music Emotion Recognition: A New Dataset, Methodology and Comparative Analysis , 2013 .

[64]  Glyn W. Humphreys The logic of mind: R.J. Nelson D. Reidel Publishing Company, London, 1982. (392 pp.) ISBN 90-277-1399-5 , 1983 .

[65]  Fabio Vignoli,et al.  Mapping Music In The Palm Of Your Hand, Explore And Discover Your Collection , 2004, ISMIR.

[66]  Y. Song,et al.  A Survey of Music Recommendation Systems and Future Perspectives , 2012 .

[67]  Markus Schedl,et al.  The neglected user in music information retrieval research , 2013, Journal of Intelligent Information Systems.

[68]  Peter Knees,et al.  Personalization in Multimodal Music Retrieval , 2011, Adaptive Multimedia Retrieval.

[69]  P. Gomez,et al.  Relationships between musical structure and psychophysiological measures of emotion. , 2007, Emotion.

[70]  Athanasios Lykartsis,et al.  The Emotionality of Sonic Events : Testing the Geneva Emotional Music Scale (GEMS) for Popular and Electroacoustic Music , 2013 .

[71]  Brigitte Röder,et al.  Crossmodal processing , 2009, Experimental Brain Research.

[72]  Eric J. Isaacson What You See Is What You Get: on Visualizing Music , 2005, ISMIR.

[73]  Tijl De Bie,et al.  Mining the Correlation between Lyrical and Audio Features and the Emergence of Mood , 2011, ISMIR.

[74]  Youngmoo E. Kim,et al.  Modeling Musical Emotion Dynamics with Conditional Random Fields , 2011, ISMIR.

[75]  Carles Fernandes Julià,et al.  SongExplorer: A Tabletop Application for Exploring Large Collections of Songs , 2009, ISMIR.

[76]  Joan Serrà,et al.  Music Mood Representations from Social Tags , 2009, ISMIR.

[77]  Bruno Gingras,et al.  Crossmodal transfer of arousal, but not pleasantness, from the musical to the visual domain. , 2012, Emotion.

[78]  Rainer Reisenzein,et al.  Experiencing activation: energetic arousal and tense arousal are not mixtures of valence and activation. , 2002, Emotion.

[79]  N. Remmington,et al.  Reexamining the circumplex model of affect. , 2000, Journal of personality and social psychology.

[80]  Josep Lluís Arcos,et al.  Visualizing and Exploring Personal Music Libraries , 2004, ISMIR.

[81]  Tuomas Eerola,et al.  The role of mood and personality in the perception of emotions represented by music , 2011, Cortex.

[82]  U. Ott,et al.  Using music to induce emotions: Influences of musical preference and absorption , 2008 .

[83]  Marcus T. Pearce,et al.  Music Cognition and the Cognitive Sciences , 2012, Top. Cogn. Sci..

[84]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[85]  György Fazekas,et al.  Design and Evaluation of Semantic Mood Models for Music Recommendation using Editorial Tags , 2013, ISMIR.

[86]  Javier Jaimovich,et al.  Emotion in Motion: A Study of Music and Affective Response , 2012, CMMR.

[87]  P. Bertelson,et al.  Multisensory integration, perception and ecological validity , 2003, Trends in Cognitive Sciences.

[88]  David A. Bulkin,et al.  Seeing sounds: visual and auditory interactions in the brain , 2006, Current Opinion in Neurobiology.

[89]  Emery Schubert Emotion felt by the listener and expressed by the music: literature review and theoretical perspectives , 2013, Front. Psychol..

[90]  Elaine Chew,et al.  Visualizing Music: Tonal Progressions and Distributions , 2007, ISMIR.

[91]  Remco C. Veltkamp,et al.  Computational Modeling of Induced Emotion Using GEMS , 2014, ISMIR.

[92]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[93]  Catherine Guastavino,et al.  User studies in the Music Information Retrieval Literature , 2011, ISMIR.

[94]  E. Glenn Schellenberg,et al.  Music Cognition: A Developmental Perspective , 2012, Top. Cogn. Sci..

[95]  J. Stephen Downie,et al.  Exploring Mood Metadata: Relationships with Genre, Artist and Usage Metadata , 2007, ISMIR.

[96]  Sergio Canazza,et al.  Kinematics-energy space for expressive interaction in music performance , 2022 .

[97]  M. Coltheart,et al.  Modularity of music processing , 2003, Nature Neuroscience.

[98]  Karrie Karahalios,et al.  Isochords: visualizing structure in music , 2007, GI '07.

[99]  Sally Jo Cunningham,et al.  Toward an understanding of the history and impact of user studies in music information retrieval , 2013, Journal of Intelligent Information Systems.

[100]  Karen B. Schloss,et al.  Music–color associations are mediated by emotion , 2013, Proceedings of the National Academy of Sciences.

[101]  S. Koelsch Towards a neural basis of music-evoked emotions , 2010, Trends in Cognitive Sciences.

[102]  Brandon G. Morton,et al.  A Comparative Study of Collaborative vs. Traditional Musical Mood Annotation , 2011, ISMIR.

[103]  Emery Schubert,et al.  Relationships between expressed and felt emotions in music , 2008 .

[104]  Jeff Pressing,et al.  Cognitive complexity and the structure of musical patterns , 1999 .

[105]  K. Scherer,et al.  Emotions evoked by the sound of music: characterization, classification, and measurement. , 2008, Emotion.

[106]  Leonard B. Meyer Emotion and Meaning in Music , 1957 .

[107]  P. Ekman An argument for basic emotions , 1992 .

[108]  C. Spence Crossmodal correspondences: A tutorial review , 2011, Attention, perception & psychophysics.

[109]  J. Sloboda,et al.  Music and emotion: Theory and research , 2001 .

[110]  Gregor Strle,et al.  Introducing a Dataset of Emotional and Color Responses to Music , 2014, ISMIR.

[111]  de Gelder Sound Enhances Visual Perception: Cross-Modal Effects of Auditory Organization on Vision , 2001 .