Natural human–robot musical interaction: understanding the music conductor gestures by using the WB-4 inertial measurement system

This paper presents an inertial measurement unit-based human gesture recognition system for a robot instrument player to understand the instructions dictated by an orchestra conductor and accordingly adapt its musical performance. It is an extension of our previous publications on natural human–robot musical interaction. With this system, the robot can understand the real-time variations in musical parameters dictated by the conductor’s movements, adding expression to its performance while being synchronized with all the other human partner musicians. The enhanced interaction ability would obviously lead to an improvement of the overall live performance, but also allow the partner musicians, as well as the conductor, to better appreciate a joint musical performance, thanks to the complete naturalness of the interaction. Graphical Abstract

[1]  Gil Weinberg,et al.  Musical interactions with a perceptual robotic percussionist , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[2]  Jonathan Harnum,et al.  Basic Music Theory: How to Read, Write, and Understand Written Music , 2001 .

[3]  Lina,et al.  Fuzzy-Appearance Manifold and Fuzzy-Nearest Distance Calculation for Model-Less 3D Pose Estimation of Degraded Face Images , 2013 .

[4]  Atsuo Takanishi,et al.  Development of anthropomorphic musical performance robots: From understanding the nature of music performance to its application to entertainment robotics , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[5]  A. Takanishi,et al.  Application of a Musical-based Interaction System to the Waseda Flutist robot WF-4RIV: Development results and performance experiments , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[6]  Constantine Stephanidis,et al.  Universal Access in Human-Computer Interaction. Applications and Services, 4th International Conference on Universal Access in Human-Computer Interaction, UAHCI 2007 Held as Part of HCI International 2007 Beijing, China, July 22-27, 2007 Proceedings, Part III , 2007, HCI.

[7]  Bharti Bansal,et al.  Gesture Recognition: A Survey , 2016 .

[8]  Eduardo Miranda,et al.  Guide to Computing for Expressive Music Performance , 2013, Springer London.

[9]  K. Saito,et al.  Musical robots: Towards a natural joint performance , 2012, 2012 First International Conference on Innovative Engineering Systems.

[10]  Atsuo Takanishi,et al.  Anthropomorphic Musical Robots Designed to Produce Physically Embodied Expressive Performances of Music , 2013, Guide to Computing for Expressive Music Performance.

[11]  Constantine Stephanidis,et al.  Universal Access in Human-Computer Interaction , 2011 .

[12]  Wolfram Burgard,et al.  Towards Service Robots for Everyday Environments - Recent Advances in Designing Service Robots for Complex Tasks in Everyday Environments , 2012, Springer Tracts in Advanced Robotics.

[13]  Guido Herrmann,et al.  Advances in Autonomous Robotics , 2012, Lecture Notes in Computer Science.

[14]  L. Bartolomeo,et al.  Development of the miniaturized wireless Inertial Measurement Unit WB-4: Pilot test for mastication analysis , 2010, 2010 IEEE/SICE International Symposium on System Integration.

[15]  Sara Weiss,et al.  Harvard Dictionary Of Music , 2016 .

[16]  Takanori Shibata,et al.  Importance of Physical Interaction between Human and Robot for Therapy , 2011, HCI.

[17]  Kenji Suzuki,et al.  Emotionally Assisted Human–Robot Interaction Using a Wearable Device for Reading Facial Expressions , 2012, Adv. Robotics.

[18]  Salvatore Sessa,et al.  Performance evaluation of the wireless inertial measurement unit WB-4 with magnetic field calibration , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[19]  Tetsuya Ogata,et al.  A Musical Robot that Synchronizes with a Coplayer Using Non-Verbal Cues , 2012, Adv. Robotics.

[20]  Gil Weinberg,et al.  The Design of a Perceptual and Improvisational Robotic Marimba Player , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[21]  Bruce A. MacDonald,et al.  HRI Evaluation of a Healthcare Service Robot , 2012, ICSR.

[22]  Javier Ramírez De La Pinta,et al.  Integration of service robots in the smart home by means of UPnP: A surveillance robot case study , 2013, Robotics Auton. Syst..

[23]  Johan A. K. Suykens,et al.  Advances in learning theory : methods, models and applications , 2003 .

[24]  Daniel Thalmann,et al.  Anthropomorphism of Artificial Agents: A Comparative Survey of Expressive Design and Motion of Virtual Characters and Social Robots , 2012, CASA 2012.

[25]  Jong-Hann Jean,et al.  Design of an entertainment robot with multimodal human-robot interactions , 2008, 2008 SICE Annual Conference.

[26]  Naomi Ehrich Leonard,et al.  Towards Human–Robot Teams: Model-Based Analysis of Human Decision Making in Two-Alternative Choice Tasks With Social Feedback , 2012, Proceedings of the IEEE.

[27]  Takatoshi Ide,et al.  Expectations for medical and healthcare robotics , 1992, Adv. Robotics.

[28]  Huosheng Hu,et al.  Use of multiple wearable inertial sensors in upper limb motion tracking. , 2008, Medical engineering & physics.

[29]  Salvatore Sessa,et al.  Online magnetic calibration of a cutting edge 9-axis wireless Inertial Measurement Unit , 2012 .

[30]  Rama Chellappa,et al.  Machine Recognition of Human Activities: A Survey , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Salvatore Sessa,et al.  Music conductor gesture recognition by using inertial measurement system for human-robot musical interaction , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[32]  Atsuo Takanishi,et al.  The waseda flutist robot No. 4 refined IV: enhancing the sound clarity and the articulation between notes by improving the design of the lips and tonguing mechanisms , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[33]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[34]  A. Conchon,et al.  Applications and services , 2001 .

[35]  Ankit Chaudhary,et al.  Intelligent Approaches to interact with Machines using Hand Gesture Recognition in Natural way: A Survey , 2011, ArXiv.

[36]  Floris Ernst,et al.  Medical Robotics , 2015, Springer International Publishing.

[37]  Makoto Hashizume,et al.  Objective Skill Evaluation for Laparoscopic Training Based on Motion Analysis , 2013, IEEE Transactions on Biomedical Engineering.

[38]  Guy Hoffman,et al.  Shimon: an interactive improvisational robotic marimba player , 2010, CHI Extended Abstracts.

[39]  Shigeki Sugano,et al.  The robot musician 'wabot-2' (waseda robot-2) , 1987, Robotics.

[40]  Atsuo Takanishi,et al.  Musical-based interaction system for the Waseda Flutist Robot , 2010, Auton. Robots.

[41]  Salvatore Sessa,et al.  A Methodology for the Performance Evaluation of Inertial Measurement Units , 2013, J. Intell. Robotic Syst..

[42]  Gil Weinberg,et al.  The Creation of a Multi-Human, Multi-Robot Interactive Jam Session , 2009, NIME.

[43]  Jong-Hwan Kim,et al.  Behavior Selection Algorithm for Personal Service Robots Using Intelligence Operating Architecture , 2012, TAROS.

[44]  Jocelyne Troccaz.  Medical Robotics: Troccaz/Medical Robotics , 2013 .

[45]  Kuo-Liang Chung,et al.  Versatile Humanoid Robots for Theatrical Performances , 2013 .