RoboCup@Home: Results in Benchmarking Domestic Service Robots

Benchmarking robotic technologies is of utmost importance for actual deployment of robotic applications in industrial and every-day environments, therefore many efforts have recently focused on this problem. Among the many different ways of benchmarking robotic systems, scientific competitions are recognized as one of the most effective ways of rapid development of scientific progress in a field. The [email protected] league targets the development and deployment of autonomous service and assistive robot technology, being essential for future personal domestic applications, and offers an important approach to benchmarking domestic and service robots. In this paper we present the new methodology for benchmarking DSR adopted in [email protected], that includes the definition of multiple benchmarks (tests) and of performance metrics based on the relationships between key abilities required to the robots and the tests. We also discuss the results of our benchmarking approach over the past years and provide an outlook on short- and mid-term goals of Home and of DSR in general.

[1]  Jean Scholtz,et al.  Using competitions to study human-robot interaction in urban search and rescue , 2005, INTR.

[2]  David P. Miller,et al.  The AAAI 1999 Mobile Robot Competitions and Exhibitions , 2000, AI Mag..

[3]  Gerhard Lakemeyer,et al.  A Robust Speech Recognition System for Service-Robotics Applications , 2008, RoboCup.

[4]  Javier Ruiz-del-Solar,et al.  Real-Time Hand Gesture Detection and Recognition Using Boosted Classifiers and Active Learning , 2007, PSIVT.

[5]  Alfredo Weitzenfeld,et al.  The Use of Scripts Based on Conceptual Dependency Primitives for the Operation of Service Mobile Robots , 2009, RoboCup.

[6]  Alexander Ferrein,et al.  Football is coming home , 2006, PCAR '06.

[7]  Hiroaki Kitano,et al.  RoboCup: A Challenge Problem for AI , 1997, AI Mag..

[8]  Roland Siegwart,et al.  International Contest for Cleaning Robots: Fun Event or a First Step towards Benchmarking Service Robots , 2003, FSR.

[9]  Jean Scholtz,et al.  Beyond Usability Evaluation: Analysis of Human-Robot Interaction at a Major Robotics Competition , 2004, Hum. Comput. Interact..

[10]  Thomas Wisspeintner,et al.  RoboCup@Home: Creating and Benchmarking Tomorrows Service Robot Applications , 2007 .

[11]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[12]  Peter Stone,et al.  Domestic Interaction on a Segway Base , 2008, RoboCup.

[13]  Luis Rueda,et al.  Advances in Image and Video Technology, Second Pacific Rim Symposium, PSIVT 2007, Santiago, Chile, December 17-19, 2007, Proceedings , 2007, PSIVT.

[14]  Thomas Deselaers,et al.  Randomized trees for real-time one-step face detection and recognition , 2008, 2008 19th International Conference on Pattern Recognition.

[15]  Daniel Schwabe,et al.  Semantic Contract Support for E-Business Processes , 2007 .

[16]  Javier Ruiz-del-Solar.  Personal Robots as Ubiquitous-Multimedial-Mobile Web Interfaces , 2007, 2007 Latin American Web Conference (LA-WEB 2007).

[17]  Hiroaki Kitano,et al.  RoboCup Rescue A Grand Challenge for Multiagent and Intelligent Systems , 2001 .

[18]  Marek P. Michalowski,et al.  Robots in the wild: observing human-robot social interaction outside the lab , 2006, 9th IEEE International Workshop on Advanced Motion Control, 2006..

[19]  Tucker R. Balch,et al.  Ten Years of the AAAI Mobile Robot Competition and Exhibition , 2002, AI Mag..

[20]  Javier Ruiz-del-Solar,et al.  Robust Object Recognition Using Wide Baseline Matching for RoboCup Applications , 2007, RoboCup.

[21]  Javier Ruiz-del-Solar,et al.  Face Recognition for Human-Robot Interaction Applications: A Comparative Study , 2009, RoboCup.