A Musical Robot that Synchronizes with a Coplayer Using Non-Verbal Cues

Music has long been used to strengthen bonds between humans. In our research, we develop musical coplayer robots with the hope that music may improve human–robot symbiosis as well. In this paper, we underline the importance of non-verbal, visual communication for ensemble synchronization at the start, during and end of a piece. We propose three cues for interplayer communication, and present a thereminplaying, singing robot that can detect them and adapt its play to a human flutist. Experiments with two naive flutists suggest that the system can recognize naturally occurring flutist gestures without requiring specialized user training. In addition, we show how the use of audio-visual aggregation can allow a robot to adapt to tempo changes quickly.

[1]  Christopher Raphael.  A Bayesian Network for Real-Time Musical Accompaniment , 2001, NIPS.

[2]  Marc Leman,et al.  Does Social Interaction Activate Music Listeners? , 2009, CMMR.

[3]  Tetsuya Ogata,et al.  Incremental polyphonic audio to score alignment using beat tracking for singer robots , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[4]  尾形 哲也,et al.  Robot Musical Accompaniment : Real-time Synchronization using Visual Cue Recognition , 2010 .

[5]  Mark B. Sandler,et al.  Phase-based note onset detection for music signals , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[6]  C. Palmer,et al.  Synchronization of Timing and Motion 435 , 2022 .

[7]  Scott S. Wiltermuth,et al.  Synchrony and Cooperation , 2009, Psychological science.

[8]  T. Chartrand,et al.  The Chameleon Effect as Social Glue: Evidence for the Evolutionary Significance of Nonconscious Mimicry , 2003 .

[9]  J. Sloboda,et al.  Spatio-Temporal Cues for Visually Mediated Synchronization , 2009 .

[10]  Katsushi Ikeuchi,et al.  Synthesis of Dance Performance Based on Analyses of Human Motion and Music , 2008 .

[11]  Hiroshi G. Okuno,et al.  Design and Implementation of Robot Audition System 'HARK' — Open Source Software for Listening to Three Simultaneous Speakers , 2010, Adv. Robotics.

[12]  Tia DeNora,et al.  :Music and Manipulation: On the Social Uses and Social Control of Music , 2007 .

[13]  Brock McElheran,et al.  Conducting Technique: For Beginners and Professionals , 1966 .

[14]  Tetsuya Ogata,et al.  Real-Time Audio-to-Score Alignment Using Particle Filter for Coplayer Music Robots , 2011, EURASIP J. Adv. Signal Process..

[15]  Marcelo M. Wanderley,et al.  The Musical Significance of Clarinetists' Ancillary Gestures: An Exploration of the Field , 2005 .

[16]  Miller Puckette,et al.  Synthetic Rehearsal: Training the Synthetic Performer , 1985, ICMC.

[17]  Roger B. Dannenberg,et al.  An On-Line Algorithm for Real-Time Accompaniment , 1984, ICMC.

[18]  Malcolm D. Macleod,et al.  Onset Detection in Musical Audio Signals , 2003, ICMC.

[19]  Tetsuya Ogata,et al.  Thereminist robot: Development of a robot theremin player with feedforward and feedback arm control based on a Theremin's pitch model , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  W. E. Fredrickson.  Band Musicians' Performance and Eye Contact as Influenced by Loss of a Visual and/or Aural Stimulus , 1994 .

[21]  Robert C. Bolles,et al.  A RANSAC-Based Approach to Model Fitting and Its Application to Finding Cylinders in Range Data , 1981, IJCAI.

[22]  Gil Weinberg,et al.  Robot-human interaction with an anthropomorphic percussionist , 2006, CHI.

[23]  Angelica LIM,et al.  Multimodal gesture recognition for robot musical accompaniment ∗ , 2010 .

[24]  Paul M. Brossier,et al.  Automatic annotation of musical audio for interactive applications , 2006 .

[25]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[26]  Nicola Knight,et al.  Rowers' high: behavioural synchrony is correlated with elevated pain thresholds , 2010, Biology Letters.

[27]  Hideki Kenmochi,et al.  VOCALOID - commercial singing synthesizer based on sample concatenation , 2007, INTERSPEECH.

[28]  T. Ogata,et al.  Robot Musical Accompaniment : Real-time Synchronization using Visual Cue Recognition , 2010 .

[29]  A. Williamon,et al.  Exploring Co-Performer Communication , 2002 .

[30]  Tetsuya Ogata,et al.  Human-robot ensemble between robot thereminist and human percussionist using coupled oscillator model , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[31]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Bob L. Sturm,et al.  A Multimodal System for Gesture Recognition in Interactive Music Performance , 2009, Computer Music Journal.

[33]  Tetsuya Ogata,et al.  Particle-filter based audio-visual beat-tracking for music robot ensemble with human guitarist , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[34]  L. Barsalou,et al.  Embodiment in Attitudes, Social Perception, and Emotion , 2005, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[35]  Kerstin Dautenhahn,et al.  What is a robot companion - friend, assistant or butler? , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[36]  Tetsuya Ogata,et al.  Integration of flutist gesture recognition and beat tracking for human-robot ensemble , 2010 .

[37]  M. Davies,et al.  Complex domain onset detection for musical signals , 2003 .

[38]  Rudolf Rasch,et al.  Timing and synchronization in ensemble performance. , 1988 .

[39]  Tetsuya Ogata,et al.  Robot musical accompaniment: integrating audio and visual cues for real-time synchronization with a human flutist , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[40]  P. Laukka,et al.  Communication of emotions in vocal expression and music performance: different channels, same code? , 2003, Psychological bulletin.

[41]  David L. Mills,et al.  Network Time Protocol (Version 3) Specification, Implementation and Analysis , 1992, RFC.

[42]  T. M. Nakra,et al.  Synthesizing Expressive Music Through the Language of Conducting , 2002 .

[43]  Hiroshi Mizoguchi,et al.  Realization of Expressive Mobile Robot , 1997, Proceedings of International Conference on Robotics and Automation.

[44]  William H. McNeill,et al.  Keeping Together in Time: Dance and Drill in Human History. , 1995 .

[45]  Roger B. Dannenberg,et al.  The Carnegie Mellon Laptop Orchestra , 2007, ICMC.

[46]  W. Andrew Schloss,et al.  On the automatic transcription of percussive music , 1985 .

[47]  Hiroshi G. Okuno,et al.  A beat-tracking robot for human-robot interaction and its evaluation , 2008, Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots.

[48]  Mark B. Sandler,et al.  A tutorial on onset detection in music signals , 2005, IEEE Transactions on Speech and Audio Processing.

[49]  Richard O. Duda,et al.  Use of the Hough transformation to detect lines and curves in pictures , 1972, CACM.