Synthetical Benchmarking of Service Robots: A First Effort on Domestic Mobile Platforms

Most of existing benchmarking tools for service robots are basically qualitative, in which a robot's performance on a task is evaluated based on completion/incompletion of actions contained in the task. In the effort reported in this paper, we tried to implement a synthetical benchmarking system on domestic mobile platforms. Synthetical benchmarking consists of both qualitative and quantitative aspects, such as task completion, accuracy of task completions and efficiency of task completions, about performance of a robot. The system includes a set of algorithms for collecting, recording and analyzing measurement data from a MoCap system. It was used as the evaluator in a competition called the BSR challenge, in which 10 teams participated, at RoboCup 2015. The paper presents our motivations behind synthetical benchmarking, the design considerations on the synthetical benchmarking system, the realization of the competition as a comparative study on performance evaluation of domestic mobile platforms, and an analysis of the teams' performance.

[1]  Justin Manzo,et al.  The DARPA Robotics Challenge [Competitions] , 2013, IEEE Robotics Autom. Mag..

[2]  Luca Iocchi,et al.  RoboCup@Home: Results in Benchmarking Domestic Service Robots , 2009, RoboCup.

[3]  Angel P. del Pobil,et al.  Benchmarks in Robotics Research , 2006 .

[4]  Tucker R. Balch,et al.  Ten Years of the AAAI Mobile Robot Competition and Exhibition , 2002, AI Mag..

[5]  C. Cobelli,et al.  A Markerless Motion Capture System to Study Musculoskeletal Biomechanics: Visual Hull and Simulated Annealing Approach , 2006, Annals of Biomedical Engineering.

[6]  Luca Iocchi,et al.  RoboCup@Home: Scientific Competition and Benchmarking for Domestic Service Robots , 2009 .

[7]  Sven Behnke,et al.  RoboCup@Home: Demonstrating Everyday Manipulation Skills in RoboCup@Home , 2012 .

[8]  Sven Behnke,et al.  Robot Competitions Ideal Benchmarks for Robotics Research , 2006 .

[9]  Domenico G. Sorrenti,et al.  Rawseeds : Building a Benchmarking Toolkit for Autonomous Robotics , 2014 .

[10]  Andrea Bonarini,et al.  Specification of General Features of Scenarios and Robots for Benchmarking Through Competitions , 2013 .

[11]  D. Holz,et al.  RoboCup@Home: Demonstrating Everyday Manipulation Skills in RoboCup@Home , 2012, IEEE Robotics & Automation Magazine.

[12]  Alan C. Schultz,et al.  The 2000 AAAI Mobile Robot Competition and Exhibition , 2001, AI Mag..

[13]  A. Bonarini,et al.  Benchmarking Through Competitions , 2013 .

[14]  Brian A. Weiss,et al.  2003 AAAI Robot Competition and Exhibition , 2004, AI Mag..

[15]  Fazel Naghdy,et al.  Motion capture in robotics review , 2009, 2009 IEEE International Conference on Control and Automation.

[16]  Michael F. P. O'Boyle,et al.  Introducing SLAMBench, a performance and accuracy benchmarking methodology for SLAM , 2014, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[17]  Katsu Yamane,et al.  Optical motion capture system with pan-tilt camera tracking and real time data processing , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[18]  Michael J. Swain,et al.  Programming CHIP for the IJCAI-95 Robot Competition , 1996, AI Mag..

[19]  Luca Iocchi,et al.  Benchmarking Intelligent Service Robots through Scientific Competitions: The RoboCup@Home Approach , 2013, AAAI Spring Symposium: Designing Intelligent Robots.