Robot musical accompaniment: integrating audio and visual cues for real-time synchronization with a human flutist

Musicians often have the following problem: they have a music score that requires 2 or more players, but they have no one with whom to practice. So far, score-playing music robots exist, but they lack adaptive abilities to synchronize with fellow players' tempo variations. In other words, if the human speeds up their play, the robot should also increase its speed. However, computer accompaniment systems allow exactly this kind of adaptive ability. We present a first step towards giving these accompaniment abilities to a music robot. We introduce a new paradigm of beat tracking using 2 types of sensory input - visual and audio - using our own visual cue recognition system and state-of-the-art acoustic onset detection techniques. Preliminary experiments suggest that by coupling these two modalities, a robot accompanist can start and stop a performance in synchrony with a flutist, and detect tempo changes within half a second.

[1]  Richard O. Duda,et al.  Use of the Hough transformation to detect lines and curves in pictures , 1972, CACM.

[2]  Robert C. Bolles,et al.  A RANSAC-Based Approach to Model Fitting and Its Application to Finding Cylinders in Range Data , 1981, IJCAI.

[3]  Roger B. Dannenberg,et al.  An On-Line Algorithm for Real-Time Accompaniment , 1984, ICMC.

[4]  W. Andrew Schloss,et al.  On the automatic transcription of percussive music , 1985 .

[5]  Miller Puckette,et al.  Synthetic Rehearsal: Training the Synthetic Performer , 1985, ICMC.

[6]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Shigeki Sugano,et al.  The robot musician 'wabot-2' (waseda robot-2) , 1987, Robotics.

[8]  Roger B. Dannenberg,et al.  Tracking Musical Beats in Real Time , 1990, ICMC.

[9]  David L. Mills,et al.  Network Time Protocol (Version 3) Specification, Implementation and Analysis , 1992, RFC.

[10]  W. E. Fredrickson.  Band Musicians' Performance and Eye Contact as Influenced by Loss of a Visual and/or Aural Stimulus , 1994 .

[11]  D L Gilden,et al.  1/f noise in human cognition. , 1995, Science.

[12]  Petri Toiviainen,et al.  An interactive MIDI accompanist , 1998 .

[13]  Shingo Uchihashi,et al.  The beat spectrum: a new approach to rhythm analysis , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[14]  Masataka Goto,et al.  An Audio-based Real-time Beat Tracking System for Music With or Without Drum-sounds , 2001 .

[15]  Christopher Raphael,et al.  Synthesizing Musical Accompaniments With Bayesian belief networks , 2001 .

[16]  Malcolm D. Macleod,et al.  Onset Detection in Musical Audio Signals , 2003, ICMC.

[17]  M. Davies,et al.  Complex domain onset detection for musical signals , 2003 .

[18]  Mark B. Sandler,et al.  Phase-based note onset detection for music signals , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[19]  Atsuo Takanishi,et al.  Development of a new anthropomorphic flutist robot WF-4 , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[20]  Gil Weinberg,et al.  Musical interactions with a perceptual robotic percussionist , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[21]  Matthew E. P. Davies,et al.  Beat Tracking Towards Automatic Musical Accompaniment , 2005 .

[22]  Marcelo M. Wanderley,et al.  The Musical Significance of Clarinetists' Ancillary Gestures: An Exploration of the Field , 2005 .

[23]  Mark B. Sandler,et al.  A tutorial on onset detection in music signals , 2005, IEEE Transactions on Speech and Audio Processing.

[24]  Paul M. Brossier,et al.  Automatic annotation of musical audio for interactive applications , 2006 .

[25]  Gil Weinberg,et al.  The design of a robotic marimba player: introducing pitch into robotic musicianship , 2007, NIME '07.

[26]  Roger B. Dannenberg,et al.  The Carnegie Mellon Laptop Orchestra , 2007, ICMC.

[27]  Hiroshi G. Okuno,et al.  A beat-tracking robot for human-robot interaction and its evaluation , 2008, Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots.

[28]  Atsuo Takanishi,et al.  Development of a real-time instrument tracking system for enabling the musical interaction with the Waseda Flutist Robot , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[29]  Gil Weinberg,et al.  Interactive jamming with Shimon: A social robotic musician , 2009, 2009 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[30]  Tetsuya Ogata,et al.  Incremental polyphonic audio to score alignment using beat tracking for singer robots , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[31]  Atsuo Takanishi,et al.  Development of a aural real-time rhythmical and harmonic tracking to enable the musical interaction with the Waseda Flutist Robot , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[32]  Tetsuya Ogata,et al.  Thereminist robot: Development of a robot theremin player with feedforward and feedback arm control based on a Theremin's pitch model , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[33]  C. Palmer,et al.  Synchronization of Timing and Motion 435 , 2022 .

[34]  Bob L. Sturm,et al.  A Multimodal System for Gesture Recognition in Interactive Music Performance , 2009, Computer Music Journal.