Metrics for performance benchmarking of multi-robot exploration

Performance benchmarking has become an important topic within robotics. It is indeed, a critical way to compare different solutions under different conditions. In this paper, we focus on performance benchmarking of multi-robot systems which explore and map unknown terrains. We present a collection of metrics to objectively compare different algorithms that can be applied to collaborative multi-robot exploration. We also identify parameters that impact robotic fleet performances. By varying the parameters, we can identify strengths and limits of an algorithm. This work is also a first concrete step to address the general problem of objectively comparing different multi-robot coordination algorithms. We illustrate these contributions with realistic simulations of the frontier-based exploration strategy. The simulations were implemented in ROS, which enables to uncouple the control software from the drivers of the robot body. We can therefore use the same code on both simulation and real robots.

[1]  Kim D. Listmann,et al.  Performance Analysis for Multi-robot Exploration Strategies , 2010, SIMPAR.

[2]  Arnoud Visser,et al.  Evaluating maps produced by urban search and rescue robots: lessons learned from RoboCup , 2009, Auton. Robots.

[3]  Adam Jacoff,et al.  Quantitative Assessment of Robot-Generated Maps , 2009 .

[4]  Nuno M. Fonseca Ferreira,et al.  Benchmark of swarm robotics distributed techniques in a search task , 2014, Robotics Auton. Syst..

[5]  Noury Bouraqadi,et al.  Distributed Constraint Reasoning Applied to Multi-robot Exploration , 2009, 2009 21st IEEE International Conference on Tools with Artificial Intelligence.

[6]  Jindong Tan,et al.  Distributed multi-robot coordination in area exploration , 2006, Robotics Auton. Syst..

[7]  Lynne E. Parker,et al.  Multiple Mobile Robot Systems , 2008, Springer Handbook of Robotics.

[8]  Anthony Stentz,et al.  Multi-robot exploration controlled by a market economy , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[9]  Andrew Howard,et al.  Multi-robot Simultaneous Localization and Mapping using Particle Filters , 2005, Int. J. Robotics Res..

[10]  Cyrill Stachniss,et al.  Robotic Mapping and Exploration , 2009, Springer Tracts in Advanced Robotics.

[11]  Francesco Amigoni,et al.  Experimental evaluation of some exploration strategies for mobile robots , 2008, 2008 IEEE International Conference on Robotics and Automation.

[12]  Raja Chatila,et al.  Performance measure for the evaluation of mobile robot autonomy , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[13]  Noury Bouraqadi,et al.  Making networked robots connectivity-aware , 2009, 2009 IEEE International Conference on Robotics and Automation.

[14]  Angel P. del Pobil,et al.  Fostering Progress in Performance Evaluation and Benchmarking of Robotic and Automation Systems [TC Spotlight] , 2014, IEEE Robotics & Automation Magazine.

[15]  Simon Lacroix,et al.  Simulating Complex Robotic Scenarios with MORSE , 2012, SIMPAR.

[16]  Wolfram Burgard,et al.  Improved Techniques for Grid Mapping With Rao-Blackwellized Particle Filters , 2007, IEEE Transactions on Robotics.

[17]  Wolfram Burgard,et al.  Collaborative multi-robot exploration , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[18]  Nicolas Jouandeau,et al.  A Survey and Analysis of Multi-Robot Coordination , 2013 .

[19]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[20]  Zhi Yan,et al.  Team Size Optimization for Multi-robot Exploration , 2014, SIMPAR.

[21]  Robert N. Lass,et al.  Metrics for Multiagent Systems , 2009 .

[22]  Angel P. del Pobil,et al.  Benchmarks in Robotics Research , 2006 .

[23]  Brian Yamauchi,et al.  Frontier-based exploration using multiple robots , 1998, AGENTS '98.