Learning from Explanations and Demonstrations: A Pilot Study

AI has become prominent in a growing number of systems, and, as a direct consequence, the desire for explainability in such systems has become prominent as well. To build explainable systems, a large portion of existing research uses various kinds of natural language technologies, e.g., text-to-speech mechanisms, or string visualizations. Here, we provide an overview of the challenges associated with natural language explanations by reviewing existing literature. Additionally, we discuss the relationship between explainability and knowledge transfer in reinforcement learning. We argue that explainability methods, in particular methods that model the recipient of an explanation, might help increasing sample efficiency. For this, we present a computational approach to optimize the learner’s performance using explanations of another agent and discuss our results in light of effective natural language explanations for humans.

[1]  Mark K. Ho,et al.  Learning Rewards from Linguistic Feedback , 2020, AAAI.

[2]  Mohamed Chetouani,et al.  Reinforcement Learning With Human Advice: A Survey , 2020, Frontiers in Robotics and AI.

[3]  Mohamed Chetouani,et al.  CLIC: Curriculum Learning and Imitation for Object Control in Nonrewarding Environments , 2019, IEEE Transactions on Cognitive and Developmental Systems.

[4]  Elmira Yadollahi,et al.  Explainable Agency by Revealing Suboptimality in Child-Robot Learning Scenarios , 2020, ICSR.

[5]  Joseph L. Austerweil,et al.  Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners , 2020, ArXiv.

[6]  Ryan J. Lowe,et al.  Learning to summarize from human feedback , 2020, NeurIPS 2020.

[7]  Tom M. Mitchell,et al.  Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations , 2020, ACL.

[8]  Ana Paiva,et al.  Explainable Agents Through Social Cues: A Review , 2020, ArXiv.

[9]  Mark O. Riedl,et al.  Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach , 2020, HCI.

[10]  Shie Mannor,et al.  Language is Power: Representing States Using Natural Language in Reinforcement Learning. , 2019, 1910.02789.

[11]  Tim Miller,et al.  Explainable Reinforcement Learning Through a Causal Lens , 2019, AAAI.

[12]  Mohamed Chetouani,et al.  Interactively shaping robot behaviour with unlabeled human instructions , 2019, Autonomous Agents and Multi-Agent Systems.

[13]  Peter Stone,et al.  Agents teaching agents: a survey on inter-agent transfer learning , 2019, Autonomous Agents and Multi-Agent Systems.

[14]  Alan Lindsay,et al.  Towards Exploiting Generic Problem Structures in Explanations for Automated Planning , 2019, K-CAP.

[15]  Shimon Whiteson,et al.  A Survey of Reinforcement Learning Informed by Natural Language , 2019, IJCAI.

[16]  Ana Paiva,et al.  Learning from a Learner , 2019, ICML.

[17]  Finale Doshi-Velez,et al.  Exploring Computational User Models for Agent Policy Summarization , 2019, IJCAI.

[18]  Scott Niekum,et al.  Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications , 2018, AAAI.

[19]  Tim Miller,et al.  Explanation in Artificial Intelligence: Insights from the Social Sciences , 2017, Artif. Intell..

[20]  Rui Liu,et al.  A review of methodologies for natural-language-facilitated human–robot cooperation , 2017, International Journal of Advanced Robotic Systems.

[21]  Daniele Magazzeni,et al.  Explainable AI Planning (XAIP): Overview and the Case of Contrastive Explanation (Extended Abstract) , 2019, Reasoning Web.

[22]  Ofra Amir,et al.  HIGHLIGHTS: Summarizing Agent Behavior to People , 2018, AAMAS.

[23]  Matthew E. Taylor.  Improving Reinforcement Learning with Human Input , 2018, IJCAI.

[24]  Mark O. Riedl,et al.  Guiding Reinforcement Learning Exploration Using Natural Language , 2017, AAMAS.

[25]  Bradley Hayes,et al.  Improving Robot Controller Transparency Through Autonomous Policy Explanation , 2017, 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI.

[26]  Karen M. Feigh,et al.  Learning From Explanations Using Sentiment and Advice in RL , 2017, IEEE Transactions on Cognitive and Developmental Systems.

[27]  Tania Lombrozo,et al.  Explanation and inference: mechanistic and functional explanations guide property generalization , 2014, Front. Hum. Neurosci..

[28]  Cynthia Breazeal,et al.  Training a Robot via Human Feedback: A Case Study , 2013, ICSR.

[29]  Brett Browning,et al.  A survey of robot learning from demonstration , 2009, Robotics Auton. Syst..

[30]  F. Keil,et al.  Explanation and understanding , 2015 .

[31]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[32]  Cynthia Breazeal,et al.  Real-Time Interactive Reinforcement Learning for Robots , 2005 .

[33]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .