Reference test courses for autonomous mobile robots

One approach to measuring the performance of intelligent systems is to develop standardized or reproducible tests. These tests may be in a simulated environment or in a physical test course. The National Institute of Standards and Technology has developed a test course for evaluating the performance of mobile autonomous robots operating in an urban search and rescue mission. The test course is designed to simulate a collapsed building structure at various levels of fidelity. The course will be used in robotic competitions, such as the American Association for Artificial Intelligence (AAAI) Mobile Robot Competition and the RoboCup Rescue. Designed to be repeatable and highly reconfigurable, the test course challenges a robot's cognitive capabilities such as perception, knowledge representation, planning, autonomy and collaboration. The goal of the test course is to help define useful performance metrics for autonomous mobile robots which, if widely accepted, could accelerate development of advanced robotic capabilities by promoting the re-use of algorithms and system components. The course may also serve as a prototype for further development of performance testing environments which enable robot developers and purchasers to objectively evaluate robots for a particular application. In this paper we discuss performance metrics for autonomous mobile robots, the use of representative urban search and rescue scenarios as a challenge domain, and the design criteria for the test course.