Performance Characterisation in Computer Vision: The Role of Statistics in Testing and Design

We consider the relationship between the performance characteristics of vision algorithms and algorithm design. In the first part we discuss the issues involved in testing. A description of good practice is given covering test objectives, test data, test metrics and the test protocol. In the second part we discuss aspects of good algorithmic design including understanding of the statistical properties of data and common algorithmic operations, and suggest how some common problems may be overcome.

[1]  Mark W. Maimone.  A Taxonomy for Stereo Computer Vision Experiments , 1996 .

[2]  Stephen J. Maybank,et al.  Probabilistic analysis of the application of the cross ratio to model based vision , 1995, International Journal of Computer Vision.

[3]  David E. Tyler,et al.  Performance Assessment by Resampling : Rigid Motion , 1998 .

[4]  Josef Kittler,et al.  Error Guided Design of a 3D Vision System , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Patrick Courtney,et al.  Algorithmic Modeling for Performance Evaluation , 1997 .

[6]  Y. J. Zhang,et al.  A survey on evaluation methods for image segmentation , 1996, Pattern Recognit..

[7]  Richard S. Stephens,et al.  Probabilistic approach to the Hough transform , 1991, Image Vis. Comput..

[8]  J. Canny.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Gerald Q. Maguire,et al.  Comparison and evaluation of retrospective intermodality brain image registration techniques. , 1997, Journal of computer assisted tomography.

[10]  Horst Bunke,et al.  Off-Line, Handwritten Numeral Recognition by Perturbation Method , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Hyeonjoon Moon,et al.  The FERET evaluation methodology for face-recognition algorithms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[12]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Pillip Greenway,et al.  Metrics for image segmentation , 1998, Defense, Security, and Sensing.

[14]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[15]  Timothy F. Cootes,et al.  Non-linear point distribution modelling using a multi-layer perceptron , 1995, Image Vis. Comput..

[16]  David E. Tyler,et al.  Performance Assessment by Resampling: Rigid Motion Estimators , 1998 .

[17]  J. L. Wayman,et al.  Best practices in testing and reporting performance of biometric devices. , 2002 .

[18]  Ihsin T. Phillips,et al.  Empirical Performance Evaluation of Graphics Recognition Systems , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Max A. Viergever,et al.  Performance Characterization in Computer Vision , 2000, Computational Imaging and Vision.

[20]  Owen Robert Mitchell,et al.  Precision Edge Contrast and Orientation Estimation , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Pascal Fua,et al.  Measuring the Self-Consistency of Stereo Algorithms , 2000, ECCV.

[22]  Neil A. Thacker,et al.  Algorithmic modelling for performance evaluation , 1997, Machine Vision and Applications.

[23]  Patrick Courtney,et al.  Databases for Performance Characterization , 1998, Theoretical Foundations of Computer Vision.

[24]  Robert M. Haralick.  Performance Characterization in Computer Vision , 1992, BMVC.

[25]  Isabelle Guyon,et al.  What Size Test Set Gives Good Error Rate Estimates? , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Jim Piper,et al.  Variability and bias in experimentally measured classifier error rates , 1992, Pattern Recognit. Lett..

[27]  Robert M. Haralick,et al.  Review and analysis of solutions of the three point perspective pose estimation problem , 1994, International Journal of Computer Vision.

[28]  Andrew W. Fitzgibbon,et al.  An Experimental Comparison of Range Image Segmentation Algorithms , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Kevin W. Bowyer,et al.  Overview of Work in Empirical Evaluation of Computer Vision Algorithms , 1998 .

[30]  Azriel Rosenfeld,et al.  Robust regression methods for computer vision: A review , 1991, International Journal of Computer Vision.

[31]  Neil A. Thacker,et al.  Statistical Analysis of a Stereo Matching Algorithm , 1992, BMVC.

[32]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .