Toward automated evaluation of interactive segmentation

We previously described a system for evaluating interactive segmentation by means of user experiments (McGuinness and O'Connor, 2010) [1]. This method, while effective, is time-consuming and labor-intensive. This paper aims to make evaluation more practicable by investigating if it is feasible to automate user interactions. To this end, we propose a general algorithm for driving the segmentation that uses the ground truth and current segmentation error to automatically simulate user interactions. We investigate four strategies for selecting which pixels will form the next interaction. The first of these is a simple, deterministic strategy; the remaining three strategies are probabilistic, and focus on more realistically approximating a real user. We evaluate four interactive segmentation algorithms using these strategies, and compare the results with our previous user experiment-based evaluation. The results show that automated evaluation is both feasible and useful.

[1]  Marie-Pierre Jolly,et al.  Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images , 2001, ICCV.

[2]  Rolf Adams,et al.  Seeded Region Growing , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Wolfgang Hörmann,et al.  Automatic Nonuniform Random Variate Generation , 2011 .

[4]  Jayaram K. Udupa,et al.  Oriented active shape models , 2006, SPIE Medical Imaging.

[5]  Hui Zhang,et al.  Image segmentation evaluation: A survey of unsupervised methods , 2008, Comput. Vis. Image Underst..

[6]  Horst Bunke,et al.  Distance Measures for Image Segmentation Evaluation , 2006, EURASIP J. Adv. Signal Process..

[7]  Alan F. Smeaton,et al.  Synchronous Collaborative Information Retrieval: Techniques and Evaluation , 2009, ECIR.

[8]  Mark O. Riedl,et al.  Toward automated exploration of interactive systems , 2002, IUI '02.

[9]  Jayaram K. Udupa,et al.  A framework for evaluating image segmentation algorithms , 2006, Comput. Medical Imaging Graph..

[10]  Begnaud Francis Hildebrand,et al.  Introduction to numerical analysis: 2nd edition , 1987 .

[11]  Aaron Fenster,et al.  Technique for evaluation of semiautomatic segmentation methods , 1999, Medical Imaging.

[12]  Emmanouil Moschidis,et al.  Simulation of user-interaction for performance evaluation of interactive image segmentation methods , 2009 .

[13]  Jayaram K. Udupa,et al.  User-Steered Image Segmentation Paradigms: Live Wire and Live Lane , 1998, Graph. Model. Image Process..

[14]  Jayaram K. Udupa,et al.  Delineation operating characteristic (DOC) curve for assessing the accuracy behavior of image segmentation algorithms , 2004, SPIE Medical Imaging.

[15]  Wim H. Hesselink,et al.  A General Algorithm for Computing Distance Transforms in Linear Time , 2000, ISMM.

[16]  Arnold W. M. Smeulders,et al.  Interaction in the segmentation of medical images: A survey , 2001, Medical Image Anal..

[17]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[18]  Tomasz Adamek,et al.  Using contour information and segmentation for object registration, modeling and retrieval , 2006 .

[19]  Claudio Gutierrez,et al.  Survey of graph database models , 2008, CSUR.

[20]  Gary Geunbae Lee,et al.  Data-driven user simulation for automated evaluation of spoken dialog systems , 2009, Comput. Speech Lang..

[21]  Raúl Rojas,et al.  SIOX: simple interactive object extraction in still images , 2005, Seventh IEEE International Symposium on Multimedia (ISM'05).

[22]  Song Wang,et al.  New benchmark for image segmentation evaluation , 2007, J. Electronic Imaging.

[23]  Emmanouil Moschidis,et al.  A systematic performance evaluation of interactive image segmentation methods based on simulated user interaction , 2010, 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[24]  Clifford Stein,et al.  Introduction to Algorithms, 2nd edition. , 2001 .

[25]  R. K. Shyamasundar,et al.  Introduction to algorithms , 1996 .

[26]  Song Wang,et al.  Image-Segmentation Evaluation From the Perspective of Salient Object Extraction , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[27]  Joel Spolsky,et al.  User Interface Design for Programmers , 2001, Apress.

[28]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[29]  Noel E. O'Connor,et al.  A comparative evaluation of interactive segmentation algorithms , 2010, Pattern Recognit..

[30]  Arnold Neumaier,et al.  Introduction to Numerical Analysis , 2001 .

[31]  Luciano da Fontoura Costa,et al.  2D Euclidean distance transform algorithms: A comparative survey , 2008, CSUR.