Benchmarking Through Competitions

Competitions are a widely used and successful tool to promote scientific and technological progress in robotics. However, their usefulness for scientific research and for successful transfer of robotic technology from laboratories to industry may be increased by introducing a scientific approach to their design, and by structuring them as benchmarking competitions, i.e., in such a way that they can also be considered as benchmarks for objective performance assessment. This is the goal of RoCKIn, an FP7 project focusing on the development of robot benchmarking competitions. I. COMPETITIONS AND BENCHMARKING The Robotics community is running hundreds of competitions every year; among them we cite the DARPA challenges, RoboCup Soccer / Rescue / @Home / @Work competitions, the ICRA Robot Challenge, the Robot Cleaning Competition, and FIRST Competitions1. The outcome of these competitions is usually a ranking among participants, highlighting the best performers. Individually and collectively, robot competitions have a very positive effect in encouraging participants to tackle challenging problems, thus promoting advances in robotics state of art. However, most robot competitions usually suffer from limitations when considered from a scientific and benchmarking perspective. For example, their results cannot be used as a benchmarking tool, which strongly reduces their potential impact as a mechanism to promote robotics to industry and limits their usefulness to research groups. Indeed, convincing companies to embrace technology originating in autonomous robotics to create new products and markets is, in fact, very difficult without established tools to assess the real-world performance of such technology and to compare different approaches. For these reasons, we believe that the times are ripe for a new way of designing robot competitions, to make them more scientifically grounded and, at the same time, more suitable for the role of objective benchmarks. The research leading to these results has received funding from the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement 601012. A list can be found at http://robots.net/rcfaq.htm. The RoCKIn project2 aims at providing tools for benchmarking through competitions to the robotics community. The aim of RoCKIn is to design and set up scientific robot competitions able to increase scientific and technological knowledge. This will require a strong attention to rigorous experimental methodologies, tempered by the need to retain the features that have made robot competitions into successful tools for the promotion of progress in robotics: first of all the “fun” and “challenge” elements that are key parts of any successful competition. RoCKIn will then exploit the scientific foundations of its robot competitions by endowing them – by design – with an additional role: that of benchmarking tools. In other terms, while the outcomes of the RoCKIn robot competitions will retain their traditional value of producing a ranking among competing solutions at competiton time, the experimental setting of the competition will also take on the more general significance of benchmarking procedures. A. From Competitions to Scientific Competitions To explore the use of competitions as benchmarking tools, it is necessary to adopt a scientific approach, investigating whether and how competitions can be treated as scientific experiments. Key differences, in fact, exist between these categories. Competitions are designed to produce a ranking at a specific moment, while scientific experiments are aimed at proving some property in a way that, once established with a successful experiment, can be considered as assessed and can be used as the starting point for further research. Experiments have the key property of being repeatable, while competitions generally cannot be repeated under exactly the same conditions. An experiment should RoCKIn (http://rockinrobotchallenge.eu/ ) started in January 2013 under the 7th Framework Programme of the European Commission. Its partners are: Associacao Do Instituto Superior Tecnico Para A Investigacao E Desenvolvimento (Portugal, Coordinator); Università degli Studi di Roma La Sapienza (Italy); Hochschule Bonn-RheinSieg (Germany); KUKA Laboratories GmbH KUKA (Germany); Politecnico di Milano (Italy); Security Challenge Limited (UK).