Assessing kinetic meaning of music and dance via deep cross-modal retrieval

Music semantics is embodied, in the sense that meaning is biologically mediated by and grounded in the human body and brain. This embodied cognition perspective also explains why music structures modulate kinetic and somatosensory perception. We leverage this aspect of cognition, by considering dance as a proxy for music perception, in a statistical computational model that learns semiotic correlations between music audio and dance video. We evaluate the ability of this model to effectively capture underlying semantics in a cross-modal retrieval task. Quantitative results, validated with statistical significance testing, strengthen the body of evidence for embodied cognition in music and show the model can recommend music audio for dance video queries and vice-versa.

[1]  Karen Bennett,et al.  The Language of Dance , 2008 .

[2]  J. Matyja.  Embodied Music Cognition: Trouble Ahead, Trouble Behind , 2016, Front. Psychol..

[3]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  C. Krumhansl,et al.  Can Dance Reflect the Structural and Expressive Qualities of Music? A Perceptual Experiment on Balanchine's Choreography of Mozart's Divertimento No. 15 , 1997 .

[5]  Minna Huotilainen,et al.  Newborn infants' auditory system is sensitive to Western music chord categories , 2013, Front. Psychol..

[6]  B. Ross,et al.  Internalized Timing of Isochronous Sounds Is Represented in Neuromagnetic Beta Oscillations , 2012, The Journal of Neuroscience.

[7]  Richard S. J. Frackowiak,et al.  The structural components of music perception. A functional anatomical study. , 1997, Brain : a journal of neurology.

[8]  Karl J. Friston,et al.  Predictive Processes and the Peculiar Case of Music , 2019, Trends in Cognitive Sciences.

[9]  S. Dehaene,et al.  Cultural Recycling of Cortical Maps , 2007, Neuron.

[10]  Nicholas Cook,et al.  Analysing Musical Multimedia , 1998 .

[11]  Yves Bestgen,et al.  Exact Expected Average Precision of the Random Baseline for System Evaluation , 2015, Prague Bull. Math. Linguistics.

[12]  R. Laban,et al.  The mastery of movement , 1950 .

[13]  R. J. Frego.  Effects of Aural and Visual Conditions on Response to Perceived Artistic Tension in Music and Dance , 1999 .

[14]  D. Moelants,et al.  Walking on music. , 2007, Human movement science.

[15]  Zohar Eitan,et al.  How music touches: Musical parameters and listeners’ audio-tactile metaphorical mappings , 2011 .

[16]  Isabelle Peretz,et al.  Tagging the Neuronal Entrainment to Beat and Meter , 2011, The Journal of Neuroscience.

[17]  Gavin M Bidelman,et al.  Brainstem correlates of behavioral and compositional preferences of musical harmony , 2011, Neuroreport.

[18]  Phil Blunsom,et al.  Multilingual Distributed Representations without Word Alignment , 2013, ICLR 2014.

[19]  Lei Chen,et al.  Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval , 2017, ACM Trans. Multim. Comput. Commun. Appl..

[20]  Karen Livescu,et al.  Multi-view Recurrent Neural Acoustic Word Embeddings , 2016, ICLR.

[21]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[22]  P. Schlenker.  Outline of Music Semantics , 2017 .

[23]  Nancy Kanwisher,et al.  Toward a universal decoder of linguistic meaning from brain activation , 2018, Nature Communications.

[24]  T. Rogers,et al.  The neural and computational bases of semantic cognition , 2016, Nature Reviews Neuroscience.

[25]  Patrik N. Juslin,et al.  What does music express? Basic emotions and beyond , 2013, Front. Psychol..

[26]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[27]  Gavin M. Bidelman,et al.  Neural Correlates of Consonance, Dissonance, and the Hierarchy of Musical Pitch in the Human Brainstem , 2009, The Journal of Neuroscience.

[28]  T. Eerola,et al.  Music Communicates Affects, Not Basic Emotions – A Constructionist Account of Attribution of Emotional Meanings to Music , 2018, Front. Psychol..

[29]  Nikoleta Popa Blanariu.  Towards a Framework of a Semiotics of Dance , 2013 .

[30]  G. Iseminger,et al.  The Aesthetics of Music , 1999 .

[31]  T. R. Knapp.  Canonical correlation analysis: A general parametric significance-testing system. , 1978 .

[32]  Ellen Winner,et al.  "Metaphorical" Mapping in Human Infants , 1981 .

[33]  R. Butler,et al.  Localization of tonal stimuli in the vertical plane. , 1968, The Journal of the Acoustical Society of America.

[34]  Jeffrey R. Binder,et al.  The Neural Career of Sensory-motor Metaphors , 2011, Journal of Cognitive Neuroscience.

[35]  Peter Kivy,et al.  The Corded Shell: Reflections on Musical Expression , 1980 .

[36]  Alan C. Evans,et al.  Neural mechanisms underlying melodic perception and memory for pitch , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[37]  M. Leman.  Embodied Music Cognition and Mediation Technology , 2007 .

[38]  M. Kiefer,et al.  Conceptual representations in mind and brain: Theoretical developments, current evidence and future directions , 2012, Cortex.

[39]  Stephen Davies,et al.  Musical Meaning and Expression , 1994 .

[40]  J. Driver,et al.  Audiovisual links in exogenous covert spatial orienting , 1997, Perception & psychophysics.

[41]  Asunción López-Varela Azcárate.  Intertextuality and Intermediality as Cross-cultural Comunication Tools: A Critical Inquiry , 2011 .

[42]  Z. Eitan,et al.  HOW MUSIC MOVES: Musical Parameters and Listeners' Images of Motion , 2006 .

[43]  M. Tervaniemi,et al.  From symbols to sounds: visual symbolic information activates sound representations. , 2004, Psychophysiology.

[44]  Steven Brown,et al.  Universals in the world’s musics , 2013 .

[45]  Marina Korsakova-Kreyn,et al.  Two-Level Model of Embodied Cognition in Music , 2018, Psychomusicology: Music, Mind, and Brain.

[46]  Alan C. Evans,et al.  Cerebellar Contributions to Motor Timing: A PET Study of Auditory and Visual Rhythm Reproduction , 1998, Journal of Cognitive Neuroscience.

[47]  中村 聡.  Analysis of music-brain interaction with simultaneous measurement of regional cerebral blood flow and electroencephalogram beta rhythm in human subjects , 2000 .

[48]  Jason M Haberman,et al.  Sensorimotor coupling in music and the psychology of the groove. , 2012, Journal of experimental psychology. General.

[49]  Marc Leman,et al.  An embodied approach to music semantics , 2010 .

[50]  M. Leman,et al.  The Role of Embodiment in the Perception of Music , 2015 .

[51]  Alexander Refsum Jensenius,et al.  Evaluating a Collection of Sound-Tracing Data of Melodic Phrases , 2018, ISMIR.

[52]  Stephen McAdams,et al.  Musical Forces and Melodic Expectations: Comparing Computer Models and Experimental Results , 2004 .

[53]  T. Wheatley,et al.  Music and movement share a dynamic structure that supports universal expressions of emotion , 2012, Proceedings of the National Academy of Sciences.

[54]  Ernst Kurth.  Ernst Kurth: Selected Writings , 1991 .

[55]  G. Lakoff.  Mapping the brain's metaphor circuitry: metaphorical thought in everyday reason , 2014, Front. Hum. Neurosci..

[56]  Irfan A. Essa,et al.  Let's Dance: Learning From Online Dance Videos , 2018, ArXiv.

[57]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Jay L. Lemke,et al.  Intertextuality and educational research , 1992 .

[59]  Jeff A. Bilmes,et al.  Deep Canonical Correlation Analysis , 2013, ICML.

[60]  Alexei A. Efros,et al.  Everybody Dance Now , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[61]  L. Trainor,et al.  Hearing what the body feels: Auditory encoding of rhythmic movement , 2007, Cognition.

[62]  George Lakoff,et al.  Explaining Embodied Cognition Results , 2012, Top. Cogn. Sci..