Competitions for Benchmarking: Task and Functionality Scoring Complete Performance Assessment

Scientific experiments and robotic competitions share some common traits that can put the debate about developing better experimental methodologies and replicability of results in robotics research on more solid ground. In this context, the Robot Competitions Kick Innovation in Cognitive Systems and Robotics (RoCKIn) project aims to develop competitions that come close to scientific experiments, providing an objective performance evaluation of robot systems under controlled and replicable conditions. In this article, by further articulating replicability into reproducibility and repeatability and by considering some results from the 2014 first RoCKIn competition, we show that the RoCKIn approach offers tools that enable the replicability of experimental results.

[1]  José Hernández-Orallo,et al.  AI Evaluation: past, present and future , 2014, ArXiv.

[2]  Pedro U. Lima,et al.  The RoCKIn@Home challenge , 2014, ISR 2014.

[3]  Pedro U. Lima,et al.  The RoCKIn@Work challenge , 2014, ISR 2014.

[4]  C. Drummond.  Replicability is not Reproducibility:Nor is it Good Science , 2009 .

[5]  Christian Collberg,et al.  Measuring Reproducibility in Computer Systems Research , 2014 .

[6]  M. Ani Hsieh,et al.  Editorial: For the JFR special issue on "Multiple collaborative field robots" , 2012, J. Field Robotics.

[7]  Odest Chadwicke Jenkins,et al.  Recasting Robotics Challenges as Experiments [Competitions] , 2011 .

[8]  Luca Iocchi,et al.  RoboCup@Home: Scientific Competition and Benchmarking for Domestic Service Robots , 2009 .

[9]  I. Hacking,et al.  Representing and Intervening. , 1986 .

[10]  Frederik Hegger,et al.  RoboCup@Work: Competing for the Factory of the Future , 2014, RoboCup.

[11]  Luca Iocchi,et al.  Benchmarking Intelligent Service Robots through Scientific Competitions: The RoboCup@Home Approach , 2013, AAAI Spring Symposium: Designing Intelligent Robots.

[12]  Thomas Dean,et al.  A Retrospective of the AAAI Robot Competitions , 1997, AI Mag..

[13]  C. Wright.  Representing and Intervening , 1985 .

[14]  Sven Behnke,et al.  Robot Competitions Ideal Benchmarks for Robotics Research , 2006 .

[15]  Bill Smart.  Competitions, Challenges, or Journal Papers? [Competitions] , 2012, IEEE Robotics Autom. Mag..

[16]  Adam Jacoff,et al.  RoboCup Rescue Robot League , 2014, RoboCup.

[17]  Thomas Bräunl,et al.  Research relevance of mobile robot competitions , 1999, IEEE Robotics Autom. Mag..

[18]  Maria Gini,et al.  Issues with Methods for Scoring Competitors in RoboCup Rescue , 2014 .