Towards Understanding Hierarchical Natural Language Commands for Robotic Navigation and Manipulation

We describe a new model for understanding hierarchical natural language commands for robot navigation and manipulation. The model has three components: a semantic structure that captures the hierarchical structure of language; a cost function that maps the command’s semantic structure to the robot’s sensorimotor capabilities; and an efficient search method for finding the lowest-cost plan. We present a proofof-concept system that carries out navigation commands in a simulated setting.