Web-Based Remote Assistance to Overcome Robot Perceptual Limitations ∗

This paper addresses the problem of overcoming visual perception limitations in service robots with remote assistance from human users. In particular, consider a scenario where a user requests the robot to perform some task that requires a perceptual ability, e.g., check if a specific mug, “my mug,” is in the lab or in an office, but the robot may not know how to recognize that object. We propose to equip the robots with the abilities to: (i) identify their own perceptual limitations, (ii) autonomously and remotely query human users for assistance, and (iii) learn new object descriptors from the interaction with humans. We successfully developed a complete initial version of our approach on our CoBot service mobile robot. The interaction with the user builds upon our previously developed semiautonomous telepresence image sharing and control. The user can now further identify the object and the robot can save the descriptor and use it in future situations. We illustrate our work with the task of learning to identify an object in the environment, and to report its presence to a user. Our ongoing work includes addressing a dynamic interaction between the robot and the remote user for visual focus of attention and different object viewing, as well as the effective storage, labeling, accessing, and sharing of multiple learned object descriptors, in particular among robots. Our goal is also to contribute the learned knowledge to crowd-robotics efforts.