Evaluation of the RoboCup Logistics League and Derived Criteria for Future Competitions

In the RoboCup Logistics League RCLL, games are governed by a semi-autonomous referee box. It also records tremendous amounts of data about state changes of the game or communication with the robots. In this paper, we analyze the data of the 2014 competition by means of Key Performance Indicators KPI. KPIs are used in industrial environments to evaluate the performance of production systems. Applying adapted KPIs to the RCLL provides interesting insights about the strategies of the robot teams. When aiming for more realistic industrial properties with a 24/7 production, where teams perform shifts without intermediate environment reset, KPIs could be a means to score the game. This could be tried first in a simulation sub-league.

[1]  Raj Madhavan,et al.  Benchmarking and standardization of intelligent robotic systems , 2009, 2009 International Conference on Advanced Robotics.

[2]  P. Nyhuis,et al.  Modeling and deriving strategic logistic measures , 2007, 2007 IEEE International Conference on Industrial Engineering and Engineering Management.

[3]  Hiroaki Kitano,et al.  RoboCup: The Robot World Cup Initiative , 1997, AGENTS '97.

[4]  Robert M. Wygant.  CLIPS - a powerful development and delivery expert system tool , 1989 .

[5]  Luca Iocchi,et al.  A unified benchmark framework for autonomous Mobile robots and Vehicles Motion Algorithms (MoVeMA benchmarks) , 2008 .

[6]  Tucker R. Balch,et al.  Ten Years of the AAAI Mobile Robot Competition and Exhibition , 2002, AI Mag..

[7]  Peter Nyhuis,et al.  Logistic Production Operating Curves – Basic Model of the Theory of Logistic Operating Curves , 2006 .

[8]  T. C. Jackson.  CIM for the process industries: a framework , 1989 .

[9]  A. Bonarini,et al.  Benchmarking Through Competitions , 2013 .

[10]  Gerhard Lakemeyer,et al.  A generic robot database and its application in fault analysis and performance evaluation , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[11]  Blai Bonet,et al.  The AIPS-98 Planning Competition , 2000, AI Mag..

[12]  Ian Horrocks,et al.  Combinations of Modal Logics , 2002, Artificial Intelligence Review.

[13]  Gerhard Lakemeyer,et al.  Simulation for the RoboCup Logistics League with Real-World Environment Agency and Multi-level Abstraction , 2014, RoboCup.

[14]  John Anderson,et al.  Robotics competitions as benchmarks for AI research , 2011, Knowl. Eng. Rev..

[15]  Peter Nyhuis,et al.  Fundamentals of Production Logistics: Theory, Tools and Applications , 2008 .

[16]  Hiroaki Kitano,et al.  RoboCup Rescue A Grand Challenge for Multiagent and Intelligent Systems , 2001 .

[17]  G. Lakemeyer,et al.  The RoboCup Logistics League as a Benchmark for Planning in Robotics , 2015 .

[18]  Andrea Bonarini,et al.  Specification of General Features of Scenarios and Robots for Benchmarking Through Competitions , 2013 .

[19]  Julian Togelius,et al.  The Mario AI Benchmark and Competitions , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[20]  E. Dahlman,et al.  A Critical Assessment of Benchmark Comparison in Planning , 2002, J. Artif. Intell. Res..

[21]  Odest Chadwicke Jenkins,et al.  Recasting Robotics Challenges as Experiments [Competitions] , 2011 .

[22]  S. Jeschke,et al.  PROPOSAL FOR ADVANCEMENTS TO THE LLSF IN 2014 AND BEYOND , 2013 .

[23]  Luca Iocchi,et al.  RoboCup@Home: Scientific Competition and Benchmarking for Domestic Service Robots , 2009 .

[24]  Hermann Lödding.  Handbook of Manufacturing Control , 2013 .