Evaluation framework for ATR algorithms

This contribution presents a comprehensive framework for algorithm evaluation. When we speak of evaluation, we have in mind that first the performance of an algorithm is measured and then the measured performance is assessed with regard to a given application. The performance assessment is done by applying an assessment function that uses desired values for the performance measures and weighting factors giving the importance of each measure, thus considering the application- specific requirements. The algorithm evaluation's goal is to verify the specification of an algorithm. This specification is mainly given by the definition of the input data and the expected output data, both of which are determined by the application. Prior to the evaluation process the algorithm specification has to be laid down by analyzing the application in order to deduce its requirements as well as by defining the application relevant data sets. To organize this sequence of preparatory steps and to formalize the accomplishment of the evaluation we have developed a 3-phase approach, consisting of the definition phase, the tuning phase, and the evaluation phase. An extensive software toolbox has been developed to support the evaluation process.