Towards a systematic assessment of the functions of unmanned autonomous systems

Being able to assess the performance of the algorithmic components of unmanned autonomous systems is a necessity. Defining repeatable and commonly shared test protocols to assess the performance of the algorithms involved in autonomy is the key to achieve standardization in the unmanned autonomous systems field (Unmanned Ground Vehicles (UGVs) and Unmanned Aerial Vehicles (UAVs)). This paper proposes a generic methodology to evaluate any function of an autonomous system and illustrates the methodology on two examples: for the evaluation of visual beacon tracking algorithm and for the evaluation of Simultaneous Localization And Mapping (SLAM) algorithms. The lessons learnt from these evaluations are then described.

[1]  Richard T. Vaughan,et al.  The Player/Stage Project: Tools for Multi-Robot and Distributed Sensor Systems , 2003 .

[2]  Sebastian Thrun,et al.  FastSLAM: a factored solution to the simultaneous localization and mapping problem , 2002, AAAI/IAAI.

[3]  Joachim Hertzberg,et al.  Ground truth evaluation of large urban 6D SLAM , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[4]  James S. Albus,et al.  Toward a Generic Model for Autonomy Levels for Unmanned Systems (ALFUS) , 2003 .

[5]  Sebastian Thrun,et al.  Robotic mapping: a survey , 2003 .

[6]  T. Collins,et al.  Occupancy grid mapping: An empirical evaluation , 2007, 2007 Mediterranean Conference on Control & Automation.

[7]  Robin Jaulmes,et al.  Towards a quantitative evaluation of simultaneous localization and mapping methods , 2009 .

[8]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[9]  S. Balakirsky,et al.  Towards Quantitative Comparisons of Robot Algorithms : Experiences with SLAM in Simulation and Real World Systems , 2007 .

[10]  Robin Jaulmes,et al.  HNG: A Robust Architecture for Mobile Robots Systems , 2008, EUROS.

[11]  D. Dufourd,et al.  A METHODOLOGY FOR ASSESSING ROBOT AUTONOMOUS FUNCTIONALITIES , 2007 .

[12]  D. Dufourd,et al.  Performance Evaluation of Road Detection and Tracking Algorithms , 2002 .

[13]  Ronald Parr,et al.  DP-SLAM: fast, robust simultaneous localization and mapping without predetermined landmarks , 2003, IJCAI 2003.

[14]  Maria L. Gini,et al.  Good Experimental Methodologies for Robotic Mapping: A Proposal , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.