Playing with the masters: A model for improvisatory musical interaction between robots and humans

We present our approach for facilitating musical interaction between robotic musicians and humans (musicians as well as non-musicians) in an improvisatory and exploratory manner. By using the iPhone as a musical instrument, we offer a quick and effective way for anyone to start creating music and communicating directly with our marimba playing robot, Shimon. Through automated style analysis of jazz masters, our system creates a novel improvisatory and interactive musical language. In a call and response exchange, the robot expands on short tunes created by human performers, in a musically rich and meaningful manner. As part of the interaction, users can also controlling the musical influences that combine to make up Shimon's improvisatory language, creating novel and inspiring responses.

[1]  Alan Dorin,et al.  Generative processes and the electronic arts , 2001, Organised Sound.

[2]  William Buxton,et al.  The Use of Hierarchy and Instance in a Data Structure for Computer Music , 1978, ICMC.

[3]  C. Krumhansl,et al.  Tracing the dynamic changes in perceived tonal organization in a spatial representation of musical keys. , 1982 .

[4]  Michael C. Mozer,et al.  Neural Network Music Composition by Prediction: Exploring the Benefits of Psychoacoustic Constraints and Multi-scale Processing , 1994, Connect. Sci..

[5]  Eric Singer,et al.  Proceedings of the 2003 Conference on New Interfaces for Musical Expression (NIME-03), Montreal, Canada LEMUR GuitarBot: MIDI Robotic String Instrument , 2022 .

[6]  Sergi Jordà.  Afasia: The Ultimate Homeric One-Man-Multimedia-Band , 2002, NIME.

[7]  Gil Weinberg,et al.  Interactive jamming with Shimon: A social robotic musician , 2009, 2009 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[8]  J. Mccormack.  Grammar-based music composition , 1996 .

[9]  C. Krumhansl.  Cognitive Foundations of Musical Pitch , 1990 .

[10]  Manuela Veloso,et al.  Bob: an improvisational music companion , 2001 .

[11]  Regan L. Mandryk,et al.  Physiological indicators for the evaluation of co-located collaborative play , 2004, CSCW.

[12]  François Pachet.  Multimedia at work - Playing with virtual musicians: the continuator in practice , 2002, IEEE MultiMedia.

[13]  Robert M. Keller,et al.  A Grammatical Approach to Automatic Improvisation , 2007 .

[14]  Guy Hoffman,et al.  Interactive improvisation with a robotic marimba player , 2011, Auton. Robots.

[15]  S. R. Holtzman,et al.  Using Generative Grammars for Music Composition , 1981 .

[16]  Gil Weinberg,et al.  ZOOZbeat: mobile music recreation , 2010, CHI EA '10.