Interactive Learning of Grounded Verb Semantics towards Human-Robot Communication

To enable human-robot communication and collaboration, previous works represent grounded verb semantics as the potential change of state to the physical world caused by these verbs. Grounded verb semantics are acquired mainly based on the parallel data of the use of a verb phrase and its corresponding sequences of primitive actions demonstrated by humans. The rich interaction between teachers and students that is considered important in learning new skills has not yet been explored. To address this limitation, this paper presents a new interactive learning approach that allows robots to proactively engage in interaction with human partners by asking good questions to learn models for grounded verb semantics. The proposed approach uses reinforcement learning to allow the robot to acquire an optimal policy for its question-asking behaviors by maximizing the long-term reward. Our empirical results have shown that the interactive learning approach leads to more reliable models for grounded verb semantics, especially in the noisy environment which is full of uncertainties. Compared to previous work, the models acquired from interactive learning result in a 48% to 145% performance gain when applied in new situations.

[1]  Shaohua Yang,et al.  Physical Causality of Action Verbs in Grounded Language Understanding , 2016, ACL.

[2]  Pat Langley,et al.  Learning hierarchical task networks by observation , 2006, ICML.

[3]  Y. Kuniyoshi,et al.  Cognitive Developmental Robotics : A Survey Article in IEEE Transactions on Autonomous Mental Development , 2009 .

[4]  Changsong Liu,et al.  Probabilistic Labeling for Efficient Referential Grounding based on Collaborative Discourse , 2014, ACL.

[5]  Luke S. Zettlemoyer,et al.  A Joint Model of Language and Perception for Grounded Attribute Learning , 2012, ICML.

[6]  Stefanie Tellex,et al.  Learning perceptually grounded word meanings from unaligned parallel data , 2012, Machine Learning.

[7]  Joyce Yue Chai,et al.  Incremental Acquisition of Verb Hypothesis Space towards Physical World Interaction , 2016, ACL.

[8]  Matthew E. Taylor,et al.  Understanding Human Teaching Modalities in Reinforcement Learning Environments: A Preliminary Report , 2011 .

[9]  Peter Stone,et al.  Learning Multi-Modal Grounded Linguistic Semantics by Playing "I Spy" , 2016, IJCAI.

[10]  Ashutosh Saxena,et al.  Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions , 2014, Int. J. Robotics Res..

[11]  Manuela M. Veloso,et al.  Teaching multi-robot coordination using demonstration of communication and state sharing , 2008, AAMAS.

[12]  Maya Cakmak,et al.  Designing robot learners that ask good questions , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[13]  Raymond J. Mooney,et al.  Unsupervised PCFG Induction for Grounded Language Learning with Highly Ambiguous Supervision , 2012, EMNLP.

[14]  Ashutosh Saxena,et al.  Environment-Driven Lexicon Induction for High-Level Instructions , 2015, ACL.

[15]  Jason Weston,et al.  Learning Through Dialogue Interactions , 2016, ArXiv.

[16]  Sonia Chernova,et al.  Interactive Hierarchical Task Learning from a Single Demonstration , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[17]  Mark Steedman,et al.  Learning STRIPS Operators from Noisy and Incomplete Observations , 2012, UAI.

[18]  Changsong Liu,et al.  Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue , 2015, AAAI.

[19]  Ann L. Brown,et al.  How people learn: Brain, mind, experience, and school. , 1999 .

[20]  Joyce Yue Chai,et al.  Collaborative Models for Referring Expression Generation in Situated Dialogue , 2014, AAAI.

[21]  Song-Chun Zhu,et al.  Jointly Learning Grounded Task Structures from Language Instruction and Visual Demonstration , 2016, EMNLP.

[22]  Geoffrey Zweig,et al.  End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning , 2016, ArXiv.

[23]  Luke S. Zettlemoyer,et al.  Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions , 2013, TACL.

[24]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[25]  Luke S. Zettlemoyer,et al.  Reinforcement Learning for Mapping Instructions to Actions , 2009, ACL.

[26]  S. Singh,et al.  Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System , 2011, J. Artif. Intell. Res..

[27]  Deb Roy,et al.  Situated Language Understanding as Filtering Perceived Affordances , 2007, Cogn. Sci..

[28]  John E. Laird,et al.  A Computational Model for Situated Task Learning with Interactive Instruction , 2016, ArXiv.

[29]  Steve J. Young,et al.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , 2006, The Knowledge Engineering Review.

[30]  Peter Stone,et al.  Learning to Interpret Natural Language Commands through Human-Robot Dialog , 2015, IJCAI.

[31]  Roberto Pieraccini,et al.  Automating spoken dialogue management design using machine learning: An industry perspective , 2008, Speech Commun..

[32]  Changsong Liu,et al.  Grounded Semantic Role Labeling , 2016, NAACL.

[33]  Yunyi Jia,et al.  Back to the Blocks World: Learning New Actions through Situated Human-Robot Dialogue , 2014, SIGDIAL Conference.

[34]  Chrystopher L. Nehaniv,et al.  Teaching robots by moulding behavior and scaffolding the environment , 2006, HRI '06.

[35]  Matthias Scheutz,et al.  Tell me when and why to do it! Run-time planner model updates via natural language instruction , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[36]  David Vandyke,et al.  On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems , 2016, ACL.

[37]  Luke S. Zettlemoyer,et al.  Learning to Parse Natural Language Commands to a Robot Control System , 2012, ISER.