Evaluating maps produced by urban search and rescue robots: lessons learned from RoboCup

This paper presents the map evaluation methodology developed for the Virtual Robots Rescue competition held as part of RoboCup. The procedure aims to evaluate the quality of maps produced by multi-robot systems with respect to a number of factors, including usability, exploration, annotation and other aspects relevant to robots and first responders. In addition to the design choices, we illustrate practical examples of maps and scores coming from the latest RoboCup contest, outlining strengths and weaknesses of our modus operandi. We also show how a benchmarking methodology developed for a simulation testbed effortlessly and faithfully transfers to maps built by a real robot. A number of conclusions may be derived from the experience reported in this paper and a thorough discussion is offered.

[1]  A. Weitzenfeld,et al.  An Omnidirectional Camera Simulation for the USARSim World , 2008 .

[2]  Francesco Amigoni,et al.  A Multi-Objective Exploration Strategy for Mobile Robots , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[3]  Stefano Carpin,et al.  Quantitative Assessments of USARSim Accuracy , 2006 .

[4]  Hans P. Moravec,et al.  Robot Evidence Grids. , 1996 .

[5]  Shane O'Sullivan,et al.  Evaluating Techniques for Resolving Redundant Information and Specularity in Occupancy Grids , 2005, Australian Conference on Artificial Intelligence.

[6]  Stefano Carpin,et al.  USARSim: a robot simulator for research and education , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[7]  Andreas Birk,et al.  The Jacobs map analysis toolkit , 2008 .

[8]  Francesco Amigoni,et al.  Experimental evaluation of some exploration strategies for mobile robots , 2008, 2008 IEEE International Conference on Robotics and Automation.

[9]  Gregory Dudek,et al.  Online control policy optimization for minimizing map uncertainty during exploration , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[10]  Francesco Amigoni,et al.  On evaluating performance of exploration strategies for an autonomous mobile robot , 2008 .

[11]  Daniel C. Asmar,et al.  A Benchmark for Outdoor Vision SLAM Systems , 2007, J. Field Robotics.

[12]  Masoud Nikravesh,et al.  Feature Extraction - Foundations and Applications , 2006, Feature Extraction.

[13]  Wolfram Burgard,et al.  Improved Techniques for Grid Mapping With Rao-Blackwellized Particle Filters , 2007, IEEE Transactions on Robotics.

[14]  Robert J. Baron,et al.  Mechanisms of Human Facial Recognition , 1981, Int. J. Man Mach. Stud..

[15]  Stefano Carpin,et al.  Bridging the Gap Between Simulation and Reality in Urban Search and Rescue , 2006, RoboCup.

[16]  Andreas Birk,et al.  Learning Geometric Concepts with an Evolutionary Algorithm , 1996, Evolutionary Programming.

[17]  Stefano Carpin,et al.  UC Mercenary Team Description Paper: RoboCup 2008 Virtual Robot Rescue Simulation League , 2008 .

[18]  Masoud Nikravesh,et al.  Feature Extraction: Foundations and Applications (Studies in Fuzziness and Soft Computing) , 2006 .

[19]  Raj Madhavan,et al.  Performance evaluation and benchmarking for next intelligent robots and systems , 2009 .

[20]  Brian A. Weiss,et al.  Test arenas and performance metrics for urban search and rescue robots , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[21]  Patrick Brézillon,et al.  Lecture Notes in Artificial Intelligence , 1999 .

[22]  Arnoud Visser,et al.  Towards heterogeneous robot teams for disaster mitigation: Results and performance metrics from RoboCup rescue , 2007, J. Field Robotics.

[23]  Jacky Baltes,et al.  A benchmark suite for mobile robots , 2000, Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113).

[24]  Hans P. Moravec.  Sensor Fusion in Certainty Grids for Mobile Robots , 1988, AI Mag..

[25]  T. Collins,et al.  Occupancy grid mapping: An empirical evaluation , 2007, 2007 Mediterranean Conference on Control & Automation.

[26]  Robin R. Murphy,et al.  Conflict Metric as a Measure of Sensing Quality , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[27]  Victor Callaghan,et al.  A benchmark for measuring mobile robot environment modelling performance , 2004, IEEE Conference on Robotics, Automation and Mechatronics, 2004..

[28]  Luca Iocchi,et al.  A unified benchmark framework for autonomous Mobile robots and Vehicles Motion Algorithms (MoVeMA benchmarks) , 2008 .

[29]  Wolfram Burgard,et al.  Exploring Unknown Environments with Mobile Robots using Coverage Maps , 2003, IJCAI.

[30]  S. Balakirsky,et al.  Towards Quantitative Comparisons of Robot Algorithms : Experiences with SLAM in Simulation and Real World Systems , 2007 .

[31]  Emanuele Frontoni,et al.  Performance metric for vision based robot localization , 2008 .

[32]  Malachy Eaton,et al.  Quantitive evaluation of sonar models and mathematical update methods for Map Building with mobile robots , 2004 .

[33]  Stefano Carpin,et al.  Where Am I? A Simulated GPS Sensor for Outdoor Robotic Applications , 2008, SIMPAR.

[34]  Andreas Birk,et al.  High Fidelity Tools for Rescue Robotics: Results and Perspectives , 2005, RoboCup.

[35]  Arnoud Visser,et al.  Towards heterogeneous robot teams for disaster mitigation: Results and performance metrics from RoboCup rescue: Field Reports , 2007 .

[36]  Jean-Christophe Baillie,et al.  RobotStadium: Online Humanoid Robot Soccer Simulation Competition , 2009, RoboCup.

[37]  Andreas Birk,et al.  Planetary Exploration in USARsim: A Case Study Including Real World Data from Mars , 2009, RoboCup.

[38]  Benjamin Kuipers,et al.  Modeling Spatial Knowledge , 1978, IJCAI.