Supervised autonomy for online learning in human-robot interaction

Abstract When a robot is learning it needs to explore its environment and how its environment responds on its actions. When the environment is large and there are a large number of possible actions the robot can take, this exploration phase can take prohibitively long. However, exploration can often be optimised by letting a human expert guide the robot during its learning. Interactive machine learning, in which a human user interactively guides the robot as it learns, has been shown to be an effective way to teach a robot. It requires an intuitive control mechanism to allow the human expert to provide feedback on the robot’s progress. This paper presents a novel method which combines Reinforcement Learning and Supervised Progressively Autonomous Robot Competencies (SPARC). By allowing the user to fully control the robot and by treating rewards as implicit, SPARC aims to learn an action policy while maintaining human supervisory oversight of the robot’s behaviour. This method is evaluated and compared to Interactive Reinforcement Learning in a robot teaching task. Qualitative and quantitative results indicate that SPARC allows for safer and faster learning by the robot, whilst not placing a high workload on the human teacher.

[1]  Brett Browning,et al.  A survey of robot learning from demonstration , 2009, Robotics Auton. Syst..

[2]  Maya Cakmak,et al.  Designing robot learners that ask good questions , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[3]  Stefan Schaal,et al.  Robot Programming by Demonstration , 2009, Springer Handbook of Robotics.

[4]  Tony Belpaeme,et al.  SPARC: an efficient way to combine reinforcement learning and supervised autonomy , 2016 .

[5]  Andrea Lockerd Thomaz,et al.  Teachable robots: Understanding human teaching behavior to build more effective robot learners , 2008, Artif. Intell..

[6]  Julie A. Adams,et al.  Critical Considerations for Human-Robot Interface Development , 2002 .

[7]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[8]  Christoph Bartneck,et al.  Anthropomorphism: Opportunities and Challenges in Human–Robot Interaction , 2014, International Journal of Social Robotics.

[9]  Tony Belpaeme,et al.  Human-Guided Learning of Social Action Selection for Robot-Assisted Therapy , 2015, MLIS@ICML.

[10]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[11]  Guy Hoffman.  OpenWoZ: A Runtime-Configurable Wizard-of-Oz Framework for Human-Robot Interaction , 2016, AAAI Spring Symposia.

[12]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[13]  Henk Nijmeijer,et al.  Robot Programming by Demonstration , 2010, SIMPAR.

[14]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[15]  Dan Olsen.  Building Interactive Systems: Principles for Human-Computer Interaction , 2009 .

[16]  Thomas G. Dietterich,et al.  Toward harnessing user feedback for machine learning , 2007, IUI '07.

[17]  Takayuki Kanda,et al.  How to train your robot - teaching service robots to reproduce human social behavior , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[18]  Clayton T. Morrison,et al.  Towards understanding how humans teach robots , 2011, UMAP'11.

[19]  Laurel D. Riek,et al.  Wizard of Oz studies in HRI , 2012, J. Hum. Robot Interact..

[20]  Peter Stone,et al.  Combining manual feedback with subsequent MDP reward signals for reinforcement learning , 2010, AAMAS.

[21]  Darya Chyzhyk,et al.  Active Learning with Bootstrapped Dendritic Classifier applied to medical image segmentation , 2013, Pattern Recognit. Lett..

[22]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[23]  Sonia Chernova,et al.  Integrating reinforcement learning with human demonstrations of varying ability , 2011, AAMAS.

[24]  Tom Ziemke,et al.  Robot-assisted therapy for autism spectrum disorders with (partially) autonomous control: Challenges and outlook , 2012, Paladyn J. Behav. Robotics.

[25]  Maya Cakmak,et al.  Power to the People: The Role of Humans in Interactive Machine Learning , 2014, AI Mag..

[26]  Miguel Angel Salichs,et al.  End-user programming of a social robot by dialog , 2011, Robotics Auton. Syst..

[27]  Tony Belpaeme,et al.  SPARC: Supervised Progressively Autonomous Robot Competencies , 2015, ICSR.

[28]  Manuela M. Veloso,et al.  Interactive Policy Learning through Confidence-Based Autonomy , 2014, J. Artif. Intell. Res..

[29]  Andreas Holzinger,et al.  Interactive machine learning for health informatics: when do we need the human-in-the-loop? , 2016, Brain Informatics.

[30]  Jerry Alan Fails,et al.  Interactive machine learning , 2003, IUI '03.