Natural Language Controlled Real-Time Object Recognition Framework for Household Robot

Assistive robot systems are designed to help individuals with physical disabilities perform routine daily living activities. Current assistive robot systems are increasingly capable of performing a variety of tasks that benefit the human user. This has created the need to specify the particular task that the disabled human user desires the system to perform. Enhanced user interfaces such as joysticks can provide fine-grained control commands, but this is time-consuming and mentally and physically taxing. This article describes an ongoing project that aims to provide people with physical disabilities a method to control a robot arm using voice commands. The goal is to enable the user to control the system using natural language, i.e., without learning a special robot control vocabulary. The work describes the design and evaluation of a real-time framework that combines speech recognition, camera-based object detection, and an inference module that matches the potentially ambiguous results of speech recognition with object detection outputs to generate a unique control input for a computer vision-based robot arm. Thus, the integration of natural language and object detection systems reduces the ambiguity in specifying tasks, one of the major bottlenecks in voice-based user interfaces. A modified version of the deep learning-based object detection network YOLO (You only look once) is used to identify all potential objects of interest in the environment. Evaluating this integrated voice and object recognition-based user interface indicates that tasks can be specified accurately in different settings.