A methodology for evaluation of task performance in robotic systems: a case study in vision-based localization

Abstract.We investigated the performance of an agent that uses visual information in a partially unknown and changing environment in a principled way. We propose a methodology to study and evaluate the performance of autonomous agents. We first analyze the system theoretically to determine the most important system parameters and to predict error bounds and biases. We then conduct an empirical analysis to update and refine the model. The ultimate goal is to develop self-diagnostic procedures. We show that although simple models can successfully predict some major effects, empirically observed performance deviates from theoretical predictions in interesting ways.