Composing HARMONI: An Open-source Tool for Human and Robot Modular OpeN Interaction

The research and development of socially interactive robots is a complex challenge because of the wide variety of capabilities needed for effective social human-robot interactions (HRI). Many of these capabilities, including perception, dialog, and control, have state of the art methods and solutions, but combining those into a comprehensive and seamless interaction is still an open challenge. We describe HARMONI, a multi-modal, open-source tool for rapid social HRI development and deployment. HARMONI is centered around a ROS package for interaction development, including decision management and node orchestration. HARMONI systematically integrates with disparate functionalities needed to conduct a meaningful social human-robot interaction such as external cloud services, AI models, and modules for sensing, planning, and acting on a variety of platforms. HARMONI was applied to the QT robot platform and usability tests were conducted to evaluate the ease and speed of development and deployment. This paper describes the architecture and design of HARMONI and reports the results of a pilot study with novice users.

[1]  Takayuki Kanda,et al.  An Interaction Design Framework for Social Robots , 2011, Robotics: Science and Systems.

[2]  Gokhan Tur,et al.  Plato Dialogue System: A Flexible Conversational AI Research Platform , 2020, ArXiv.

[3]  Guy Hoffman,et al.  Blossom: A Handcrafted Open-Source Robot , 2019 .

[4]  Maja J. Matarić,et al.  Towards Autonomous Moderation of an Assembly Game , 2016, RO-MAN 2016.

[5]  Bilge Mutlu,et al.  Embodiment in Socially Interactive Robots , 2019, Found. Trends Robotics.

[6]  Thilo Michael.  Retico: An incremental framework for spoken dialogue systems , 2020, SIGdial.

[7]  Petter Ögren,et al.  Behavior Trees in Robotics and AI: An Introduction , 2017, ArXiv.

[8]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[9]  Milan Sonka,et al.  Image Processing, Analysis and Machine Vision , 1993, Springer US.

[10]  Fabrizio Montesi,et al.  Microservices: Yesterday, Today, and Tomorrow , 2017, Present and Ulterior Software Engineering.

[11]  Jugal K. Kalita,et al.  A Survey of the Usages of Deep Learning for Natural Language Processing , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Emilia I. Barakova,et al.  End-user programming architecture facilitates the uptake of robots in social therapies , 2013, Robotics Auton. Syst..

[13]  Bruno Maisonnier,et al.  Choregraphe: a graphical tool for humanoid robot programming , 2009, RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication.

[14]  E.E. Pissaloux,et al.  Image Processing , 1994, Proceedings. Second Euromicro Workshop on Parallel and Distributed Processing.

[15]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[16]  Maja J. Mataric,et al.  An architecture for rehabilitation task practice in socially assistive human-robot interaction , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[17]  Dirk Merkel,et al.  Docker: lightweight Linux containers for consistent development and deployment , 2014 .

[18]  Alexander Werner,et al.  A Framework for Human-Robot Interaction User Studies , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[19]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[20]  Takayuki Kanda,et al.  Human-robot interaction design using Interaction Composer eight years of lessons learned , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[21]  Robert C. Martin.  Clean Code - a Handbook of Agile Software Craftsmanship , 2008 .

[22]  R. de Mori.  Spoken language understanding: a survey , 2007, 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU).

[23]  Xipeng Qiu,et al.  Pre-trained models for natural language processing: A survey , 2020, Science China Technological Sciences.

[24]  Gregory M. P. O'Hare,et al.  The Social Robot Architecture: A Framework for Explicit Social Interaction , 2005 .

[25]  Nikolaos Doulamis,et al.  Deep Learning for Computer Vision: A Brief Review , 2018, Comput. Intell. Neurosci..

[26]  Enrique Coronado,et al.  Visual Programming Environments for End-User Development of intelligent and social robots, a systematic review , 2020, J. Comput. Lang..

[27]  Antonio Cisternino,et al.  Designing the Mind of a Social Robot , 2018 .

[28]  Emilia I. Barakova,et al.  User-Friendly Robot Environment for Creation of Social Scenarios , 2011, IWINAC.

[29]  Sotiris Makris,et al.  Human–robot interaction review and challenges on task planning and programming , 2016, Int. J. Comput. Integr. Manuf..

[30]  Tony Belpaeme,et al.  Child-Robot Interaction: Perspectives and Challenges , 2013, ICSR.

[31]  Yorick Wilks,et al.  Natural Language Understanding Systems Within The AI Paradigm: A Survey And Some Comparisons , 1974, ACL Microfiche Series 1-83, Including Computational Linguistics.

[32]  Frederic H. K. Siepmann.  Behavior coordination for reusable system design in interactive robotics , 2013 .

[33]  J. Glasby.  All together now... , 2003, Nature Reviews Microbiology.

[34]  Thomas Groechel,et al.  Using Socially Expressive Mixed Reality Arms for Enhancing Low-Expressivity Robots , 2019, 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).

[35]  Kerstin Dautenhahn,et al.  Socially intelligent robots: dimensions of human–robot interaction , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[36]  Sudhir N. Dhage,et al.  Survey on Virtual Assistant: Google Assistant, Siri, Cortana, Alexa , 2018, Communications in Computer and Information Science.

[37]  Bilge Mutlu,et al.  Robot behavior toolkit: Generating effective social behaviors for robots , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[38]  S. Camille Peres,et al.  Validation of the System Usability Scale (SUS) , 2013 .

[39]  Stephen R. Schach,et al.  Object-oriented and classical software engineering , 1995 .

[40]  Sean Andrist,et al.  Rapid development of multimodal interactive systems: a demonstration of platform for situated intelligence , 2017, ICMI.

[41]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[42]  Allison Sauppé,et al.  Design patterns for exploring and prototyping human-robot interactions , 2014, CHI.

[43]  David McNeill,et al.  rrSDS: Towards a Robot-ready Spoken Dialogue System , 2020, SIGDIAL.