Towards a Gesture-Sound Cross-Modal Analysis

This article reports on the exploration of a method based on canonical correlation analysis (CCA) for the analysis of the relationship between gesture and sound in the context of music performance and listening. This method is a first step in the design of an analysis tool for gesture-sound relationships. In this exploration we used motion capture data recorded from subjects performing free hand movements while listening to short sound examples. We assume that even though the relationship between gesture and sound might be more complex, at least part of it can be revealed and quantified by linear multivariate regression applied to the motion capture data and audio descriptors extracted from the sound examples. After outlining the theoretical background, the article shows how the method allows for pertinent reasoning about the relationship between gesture and sound by analysing the data sets recorded from multiple and individual subjects.

[1]  Alexander Refsum Jensenius,et al.  Playing "Air Instruments": Mimicry of Sound-Producing Gestures by Novices and Experts , 2005, Gesture Workshop.

[2]  Stefan Kopp,et al.  Co-expressivity of Speech and Gesture: Lessons for Models of Aligned Speech and Gesture Production , 2007 .

[3]  Marina Basu.  The Embodied Mind: Cognitive Science and Human Experience , 2004 .

[4]  K. Scherer,et al.  Multimodal expression of emotion: affect programs or componential appraisal patterns? , 2007, Emotion.

[5]  A. Berthoz,et al.  Le sens du mouvement , 1997 .

[6]  Rolph E. Anderson,et al.  Multivariate Data Analysis (7th ed. , 2009 .

[7]  G. Luck,et al.  Ensemble Musicians’ Synchronization With Conductors’ Gestures: An Automated Feature-Extraction Analysis , 2006 .

[8]  Antonio Camurri,et al.  Gesture-Based Communication in Human-Computer Interaction , 2003, Lecture Notes in Computer Science.

[9]  Michael Elad,et al.  Pixels that sound , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[10]  M. Leman.  Embodied Music Cognition and Mediation Technology , 2007 .

[11]  Egil Haga,et al.  Correspondences between music and body movement , 2008 .

[12]  Sotaro Kita,et al.  What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking , 2003 .

[13]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents: Research Articles , 2004 .

[14]  Alva Noë,et al.  Action in Perception , 2006, Representation and Mind.

[15]  T. Metzinger,et al.  The emergence of a shared action ontology: Building blocks for a theory , 2003, Consciousness and Cognition.

[16]  Antonio Camurri,et al.  Recognizing emotion from dance movement: comparison of spectator recognition and automated techniques , 2003, Int. J. Hum. Comput. Stud..

[17]  G. Rizzolatti,et al.  Hearing Sounds, Understanding Actions: Action Representation in Mirror Neurons , 2002, Science.

[18]  E. Large.  On synchronizing movements to music , 2000 .

[19]  Nicolas Courty,et al.  Gesture in Human-Computer Interaction and Simulation , 2006 .

[20]  Sofia Dahl,et al.  Expressiveness of Musician's Body Movements in Performances on Marimba , 2003, Gesture Workshop.

[21]  Marcelo M. Wanderley,et al.  Music and Motion: How Music-Related Ancillary Body Movements Contribute to the Experience of Music , 2009 .

[22]  J. Hair.  Multivariate data analysis , 1972 .

[23]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents , 2004, Comput. Animat. Virtual Worlds.

[24]  Marcelo M. Wanderley,et al.  Trends in Gestural Control of Music , 2000 .

[25]  D. Moelants,et al.  Walking on music. , 2007, Human movement science.

[26]  Alexander Refsum Jensenius,et al.  Action-sound : developing methods and tools to study music-related body movement , 2007 .

[27]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[28]  Frédéric Bevilacqua,et al.  Gesture Analysis of Violin Bow Strokes , 2005, Gesture Workshop.

[29]  M. Wiesendanger,et al.  Music, motor control and the brain , 2006 .