Evaluation metric for image understanding

We propose in this paper a new evaluation metric that enables to quantify the quality of an image interpretation result. This metric takes into account the a priori knowledge used by the interpretation algorithm and the ground truth associated with the original image. We combine two metrics that evaluate the localization and recognition results of each detected object. We show that the proposed metric fulfills some theoretical properties and has a correct behavior face to empirical experiments on an image benchmark database. We think that this metric could be a reliable reference for image and video understanding competitions.

[1]  Baptiste Hemery,et al.  Evaluation Protocol for Localization Metrics , 2008, ICISP.

[2]  Rita Cucchiara,et al.  Detecting Moving Objects, Ghosts, and Shadows in Video Streams , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[4]  Cordelia Schmid,et al.  Scale-invariant shape features for recognition of object categories , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[5]  E. D'Angelo,et al.  ROBIN Challenge Evaluation principles and metrics , 2006 .

[6]  L. Vinet.  Segmentation et mise en correspondance de régions de paires d'images stéréoscopiques , 1991 .

[7]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[8]  Cordelia Schmid,et al.  The 2005 PASCAL Visual Object Classes Challenge , 2005, MLCW.

[9]  Luc Van Gool,et al.  The 2005 PASCAL Visual Object Classes Challenge , 2005, MLCW.

[10]  Ihsin T. Phillips,et al.  Empirical Performance Evaluation of Graphics Recognition Systems , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Jean-Michel Jolion,et al.  Object count/area graphs for the evaluation of object detection and segmentation algorithms , 2006, International Journal of Document Analysis and Recognition (IJDAR).

[12]  C. Schmid,et al.  Scale-invariant shape features for recognition of object categories , 2004, CVPR 2004.

[13]  Jianguo Zhang,et al.  The PASCAL Visual Object Classes Challenge , 2006 .

[14]  J. Munkres.  ALGORITHMS FOR THE ASSIGNMENT AND TRANSIORTATION tROBLEMS* , 1957 .

[15]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[16]  Baptiste Hemery,et al.  Evaluation Protocol for Localization Metrics Application to a Comparative Study , 2008 .