A general framework for the evaluation of symbol recognition methods

Performance evaluation is receiving increasing interest in graphics recognition. In this paper, we discuss some questions regarding the definition of a general framework for evaluation of symbol recognition methods. The discussion is centered on three key elements in performance evaluation: test data, evaluation metrics and protocols of evaluation. As a result of this discussion we state some general principles to be taken into account for the definition of such a framework. Finally, we describe the application of this framework to the organization of the first contest on symbol recognition in GREC’03, along with the results obtained by the participants.

[1]  Y. J. Zhang,et al.  A survey on evaluation methods for image segmentation , 1996, Pattern Recognit..

[2]  Ihsin T. Phillips,et al.  The Second International Graphics Recognition Contest - Raster to Vector Conversion: A Report , 1997, GREC.

[3]  Atul K. Chhabra,et al.  Symbol Recognition : An Overview , 2005 .

[4]  Mario Vento,et al.  Symbol recognition in documents: a collection of techniques? , 2000, International Journal on Document Analysis and Recognition.

[5]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[6]  Dov Dori,et al.  A protocol for performance evaluation of line detection algorithms , 1997, Machine Vision and Applications.

[7]  N. A. Thacker,et al.  Performance Characterisation in Computer Vision: The Role of Statistics in Testing and Design , 2003 .

[8]  Xavier Hilaire.  A matching scheme to enhance performance evaluation of raster-to-vector conversion algorithms , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[9]  Dov Dori,et al.  Extended Summary of the Arc Segmentation Contest , 2001, GREC.

[10]  Isabelle Guyon,et al.  DATA SETS FOR OCR AND DOCUMENT IMAGE UNDERSTANDING RESEARCH , 1997 .

[11]  Ernest Valveny,et al.  Performance Evaluation of Symbol Recognition , 2004, Document Analysis Systems.

[12]  Robert M. Haralick,et al.  Performance characterization in image analysis: thinning, a case in point , 1992, Pattern Recognit. Lett..

[13]  Hyeonjoon Moon,et al.  The FERET Evaluation Methodology for Face-Recognition Algorithms , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Henry S. Baird,et al.  The State of the Art of Document Image Degradation Modelling , 2007 .

[15]  Debashis Ghosh,et al.  An analytic approach for generation of artificial hand-printed character database from given generative models , 1999, Pattern Recognit..

[16]  Basilios Gatos,et al.  ICDAR 2003 page segmentation competition , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[17]  Dorothea Blostein,et al.  Graphics Recognition Algorithms and Applications , 2002, Lecture Notes in Computer Science.

[18]  Jacques Labiche,et al.  Contextual System of Symbol Structural Recognition based on an Object-Process Methodology , 2005 .

[19]  Robert M. Haralick,et al.  A Statistical, Nonparametric Methodology for Document Degradation Model Validation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Rama Chellappa,et al.  Design, Integration, and Evaluation of Form-Based Handprint and OCR Systems | NIST , 1996 .

[21]  Robert M. Haralick,et al.  Document Degradation Models: Parameter Estimation and Model Validation , 1994, MVA.

[22]  Patrick Courtney,et al.  Databases for Performance Characterization , 1998, Theoretical Foundations of Computer Vision.

[23]  Basilios Gatos,et al.  ICDAR2005 page segmentation competition , 2007, Eighth International Conference on Document Analysis and Recognition (ICDAR'05).

[24]  Horst Bunke,et al.  Handbook of Character Recognition and Document Image Analysis , 1997 .

[25]  Karl Tombre,et al.  Graphics Recognition Algorithms and Systems , 1997, Lecture Notes in Computer Science.

[26]  Dov Dori,et al.  Graphics recognition - algorithms and systems - general conclusions , 1998 .

[27]  Daniel P. Lopresti,et al.  Issues in Ground-Truthing Graphic Documents , 2001, GREC.

[28]  Robert M. Haralick,et al.  Performance Characterization in Computer Vision , 1993, BMVC.

[29]  Ihsin T. Phillips,et al.  Empirical Performance Evaluation of Graphics Recognition Systems , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Jacques Blanc-Talon,et al.  Imaging and vision systems: theory, assessment and applications , 2001 .

[31]  Ernest Valveny,et al.  Symbol Recognition: Current Advances and Perspectives , 2001, GREC.

[32]  Ming Ye,et al.  Algorithm performance contest , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[33]  Jin Hyeong Park,et al.  Performance evaluation of object detection algorithms , 2002, Object recognition supported by user interaction for service robots.

[34]  S. Lucas,et al.  ICDAR 2003 robust reading competitions: entries, results, and future directions , 2005, International Journal of Document Analysis and Recognition (IJDAR).