Virtual Tools for Supervisory and Collaborative Control of Robots

Often, robotics has failed to meet industry expectations because programming robots is tedious, requires specialists, and often does not provide enough real flexibility to be worth the investment. In order to advance beyond a possible robotics plateau, an integrating technology will need to emerge that can take advantage of complex new robotic capabilities while making systems easier for nonrobotics people to use. This research introduces virtual tools with robotic attributes, and collaborative control concepts, that enable experts in areas other than robotics to simply point and direct sophisticated robots and machines to do new tasks. A system of robots that are directed using such virtual tools is now in place at the Pennsylvania State University (Penn State) and has been replicated at Sandia National Laboratories. (Mpeg movies from the Penn State Virtual Tools and Robotics Laboratory are at http://virtuoso.psu.edu/ mpeg_page.html.) Virtual tools, which appear as graphic representations of robot endeffectors interwoven into live video, carry robotic attributes that define trajectory details and determine how to interpret sensor readings for a particular type of task. An operator, or team of experts, directs robot tasks by virtually placing these tool icons in the scene. The operator(s) direct tasks involving attributes in the same natural way that supervisors direct human subordinates to, for example, put that there, dig there, cut there, and grind there. In this human-machine interface, operators do not teach entire tasks via virtual telemanipulation. They define key action points. The virtual tool attributes allow operators to stay at a supervisory level, doing what humans can do best in terms of task perceptualization, while robots plan appropriate trajectories and a variety of tool-dependent executions. Neither the task experts (e.g., in hazardous environments) nor the plant supervisors (e.g., in remote manufacturing applications) must turn over control to specialized robot technicians for long periods. Within this concept, shutting down a plant to reprogram robots to produce a new product, for example, is no longer required. Further, even though several key collaborators may be in different cities for a particular application, they may work with other experts over a project net that is formed for a particular mission. (We link simply by sending video frames over Netscape.) Using a shared set of virtual tools displayed simultaneously on each of the collaborator workstations, experts virtually enter a common videographic scene to direct portions of a task while graphically and verbally discussing alternatives with the other experts. In the process of achieving collaborative consensus, the robots are automatically programmed as a byproduct of using the virtual tools to decide what should be done and where. The robots can immediately execute the task for all to see once consensus is reached. Virtual tools and their attributes achieve robotic flexibility without requiring specialized programming or telemanipulation on the part of in situ operators. By sharing the virtual tools over project nets, noncollocated experts may now contribute to robot and intelligent machine tasks. To date, we have used virtual tools to direct a large gantry robot at Sandia National Laboratories from Penn State. We will soon have multiple collaborators sharing the virtual tools remotely, with a protocol for participants to take turns placing and moving virtual tools to define portions of complex tasks in other industrial, space-telerobotic, and educational environments. Attributes from each area of robotics research are envisioned with virtual tools as a repository for combining these independently developed robotic capabilities into integrated entities that are easy for an operator to understand, use, and modify.