Algorithmic modelling for performance evaluation

Many of the machine vision algorithms described in the literature are tested on a very small number of images. It is generally agreed that algorithms need to be tested on much larger numbers if any statistically meaningful measure of performance is to be obtained. However, these tests are rarely performed; in our opinion this is normally due to two reasons. Firstly, the scale of the testing problem is daunting when high levels of reliability are sought, since it is the proportion of failure cases that allows the reliability to be assessed and a large number of failure cases are needed to form an accurate estimation of reliability. For reliable and robust algorithms, this requires an inordinate number of test cases. Secondly, there is the difficulty of selecting test images to ensure that they are representative. This is aggravated by the fact that the assumptions made may be valid in one application domain but not in another. Hence, it is very difficult to relate the results of one evaluation to other users’ requirements. While it is true that published papers in the vision area must contain some evidence of the successful application of the suggested technique, a whole host of reasons have been put forward as to why researchers do not attempt to evaluate their algorithms more rigorously. These objections are valid only within a closely defined context and do not stand up to critical examination [13]. The real problem seems to be the time required for the various stages of algorithm development. The ratiotheory: implementation: evaluation seems to scale according to the rule of thumb 1 : 10 : 100 [13]. The effort required to get a new idea published is thus far less than an extensive empirical evaluation, which is a considerable demotivation for researchers to do evaluation work, particularly as evaluation is not much valued as publishable material in either conferences or journals. However, the truth of the matter is that unless algorithms are evaluated – and in a manner that can be used to predict the capabilities of a technique on an unseen data set – it is unlikely to be re-implemented and used. Moreover, the subject cannot advance without a well-founded scientific methodology, which it will not have without an acknowledged system for