An Artificial Intelligence Architecture for Musical Expressiveness that Learns by Imitation

Interacting with musical avatars have been increasingly popular over the years, with the introduction of games like Guitar Hero and Rock Band. These games provide MIDIequipped controllers that look like their real-world counterparts (e.g. MIDI guitar, MIDI drumkit) that the users play to control their designated avatar in the game. The performance of the user is measured against a score that needs to be followed. However, the avatar does not move in response to how the user plays, it follows some predefined movement pattern. If the user plays badly, the game ends with the avatar ending the performance (i.e. throwing the guitar on the floor). The gaming experience would increase if the avatar would move in accordance with user input. This paper presents an architecture that couples musical input with body movement. Using imitation learning, a simulated human robot learns to play the drums like human drummers do, both visually and auditory. Learning data is recorded using MIDI and motion tracking. The system uses an artificial intelligence approach to implement imitation learning, employing artificial neural networks.

[1]  G. Rizzolatti,et al.  Premotor cortex and the recognition of motor actions. , 1996, Brain research. Cognitive brain research.

[2]  Michael I. Jordan,et al.  Forward Models: Supervised Learning with a Distal Teacher , 1992, Cogn. Sci..

[3]  Gerhard Widmer,et al.  Learning to Play Like the Great Pianists , 2005, IJCAI.

[4]  A. Meltzoff,et al.  Imitation of Facial and Manual Gestures by Human Neonates , 1977, Science.

[5]  Maja J. Matarić,et al.  Sensory-motor primitives as a basis for imitation: linking perception to action and biology to robotics , 2002 .

[6]  Yiannis Demiris,et al.  Hierarchical attentive multiple models for execution and recognition of actions , 2006, Robotics Auton. Syst..

[7]  Christopher Raphael,et al.  Orchestra in a Box: A System for Real-Time Musical Accompaniment , 2003, IJCAI 2003.

[8]  Sorin Moga,et al.  From Perception-Action Loops to Imitation Processes: A Bottom-Up Approach of Learning by Imitation , 1998, Appl. Artif. Intell..

[9]  Stefan Schaal,et al.  Is imitation learning the route to humanoid robots? , 1999, Trends in Cognitive Sciences.

[10]  Yiannis Demiris,et al.  Groovy Neural Networks , 2008, ECAI.

[11]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[12]  Alexander Refsum Jensenius,et al.  Proceedings of the International Conference on New Interfaces for Musical Expression , 2011 .

[13]  K. Dautenhahn,et al.  Imitation in Animals and Artifacts , 2002 .

[14]  Norman I. Badler,et al.  Real-Time Inverse Kinematics of the Human Arm , 1996, Presence: Teleoperators & Virtual Environments.

[15]  François Pachet.  Interacting with a Musical Learning System: The Continuator , 2002, ICMAI.

[16]  Aude Billard,et al.  DRAMA, a Connectionist Architecture for Control and Learning in Autonomous Robots , 1999, Adapt. Behav..

[17]  Yiannis Demiris,et al.  A Groovy Virtual Drumming Agent , 2009, IVA.

[18]  J. Piaget.  Play, dreams and imitation in childhood , 1951 .

[19]  A. Goldman,et al.  Mirror neurons and the simulation theory of mind-reading , 1998, Trends in Cognitive Sciences.

[20]  John Shawe-Taylor,et al.  Using string kernels to identify famous performers from their playing style , 2004, Intell. Data Anal..

[21]  Axel Tidemann,et al.  Using Multiple Models to Imitate Drumming , 2010 .

[22]  Ramón López de Mántaras,et al.  Ai and Music: From Composition to Expressive Performance , 2002, AI Mag..

[23]  K. Dautenhahn,et al.  The correspondence problem , 2002 .

[24]  Maarten Grachten,et al.  A case based approach to expressivity-aware tempo transformation , 2006, Machine Learning.

[25]  Angelo Cangelosi,et al.  An Embodied Model for Sensorimotor Grounding and Grounding Transfer: Experiments With Epigenetic Robots , 2006, Cogn. Sci..

[26]  D. Wolpert,et al.  Internal models in the cerebellum , 1998, Trends in Cognitive Sciences.

[27]  M. Kawato,et al.  Behavioral/systems/cognitive Functional Magnetic Resonance Imaging Examination of Two Modular Architectures for Switching Multiple Internal Models , 2022 .

[28]  Dan Gusfield.  Algorithms on Strings, Trees, and Sequences - Computer Science and Computational Biology , 1997 .

[29]  Alfonso Caramazza,et al.  Asymmetric fMRI adaptation reveals no evidence for mirror neurons in humans , 2009, Proceedings of the National Academy of Sciences.

[30]  Jun Tani,et al.  Self-organization of distributedly represented multiple behavior schemata in a mirror system: reviews of robot experiments using RNNPB , 2004, Neural Networks.