Performance Evaluation of Image Processing Algorithms in CADe

One of the key limitations of existing image processing algorithms for computer-aided detection (CADe) is that they are often designed and evaluated in an ad hoc manner. This paper characterizes some of the issues and shortcomings in existing performance evaluation paradigms for image processing algorithms in breast cancer screening, particularly in the context of computer aided detection. We present the framework for establishing a performance evaluation process using standardized criteria. We conclude with some specific recommendations to improve the infrastructure for evaluation the performance of image processing algorithms.

[1]  William M. Wells,et al.  Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation , 2004, IEEE Transactions on Medical Imaging.

[2]  Paul S. Fisher,et al.  Image quality measures and their performance , 1995, IEEE Trans. Commun..

[3]  E. R. Davies.  Tradeoffs between speed and accuracy in two-stage template matching , 1988 .

[4]  Daniel Rueckert,et al.  Comparison and evaluation of rigid and nonrigid registration of breast MR images , 1999, Medical Imaging.

[5]  C. Nodine,et al.  Nature of expertise in searching mammograms for breast masses , 1996, Medical Imaging.

[6]  S. Ciatto,et al.  Comparison of standard reading and computer aided detection (CAD) on a national proficiency test of screening mammography. , 2003, European journal of radiology.

[7]  H P Chan,et al.  Combined adaptive enhancement and region-growing segmentation of breast masses on digitized mammograms. , 1999, Medical physics.

[8]  Derek L. G. Hill,et al.  Validation of Non-rigid Registration of Contrast-Enhanced MR Mammography Using Finite Element Methods , 2002 .

[9]  Gerard J. Holzmann,et al.  The logic of bugs , 2002, SIGSOFT '02/FSE-10.

[10]  Nico Karssemeijer,et al.  Discrete dynamic contour model for mass segmentation in digital mammograms , 1999, Medical Imaging.

[11]  Berkman Sahiner,et al.  Sensitivity of noncommercial computer-aided detection system for mammographic breast cancer detection: pilot clinical trial. , 2004, Radiology.

[12]  Carolyn Kimme-Smith.  Clinical considerations for a mammography database , 1993, Electronic Imaging.

[13]  J. Baker,et al.  Computer-aided detection (CAD) in screening mammography: sensitivity of commercial CAD systems for detecting architectural distortion. , 2003, AJR. American journal of roentgenology.

[14]  Stephen M. Smith,et al.  A Non-Rigid Registration Algorithm for Dynamic Breast MR Images , 1999, Artif. Intell..

[15]  M A Helvie,et al.  Linear motion correction in three dimensions applied to dynamic gadolinium enhanced breast imaging. , 1999, Medical physics.

[16]  D. Rueckert,et al.  Comparison and evaluation of rigid, affine, and nonrigid registration of breast MR images. , 1999, Journal of computer assisted tomography.

[17]  Martin R. Woodward,et al.  Insights into software testing , 2001, Softw. Focus.

[18]  E. Thurfjell,et al.  Benefit of independent double reading in a population-based mammography screening program. , 1994, Radiology.

[19]  G. P. Cohen,et al.  Characteristics of breast carcinomas missed by screening radiologists. , 1997, Radiology.

[20]  M. Masotti,et al.  A novel featureless approach to mass detection in digital mammograms based on support vector machines. , 2004, Physics in medicine and biology.

[21]  Neil A. Thacker,et al.  Algorithmic modelling for performance evaluation , 1997, Machine Vision and Applications.

[22]  Nico Karssemeijer,et al.  ADAPTIVE NOISE EQUALIZATION AND RECOGNITION OF MICROCALCIFICATION CLUSTERS IN MAMMOGRAMS , 1993 .

[23]  K Doi,et al.  Automated segmentation of digitized mammograms. , 1995, Academic radiology.

[24]  Maryellen L. Giger,et al.  Automated seeded lesion segmentation on digital mammograms , 1998, IEEE Transactions on Medical Imaging.

[25]  O. Nevalainen,et al.  Accurate segmentation of the breast region from digitized mammograms. , 2001, Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society.

[26]  Rangaraj M. Rangayyan,et al.  Region-based contrast enhancement of mammograms , 1992, IEEE Trans. Medical Imaging.

[27]  C. Vyborny,et al.  Breast cancer: importance of spiculation in computer-aided detection. , 2000, Radiology.

[28]  M. A. Wirth,et al.  An algorithm for extracting the skin surface in MR breast images , 2002 .

[29]  K. J. Ray Liu,et al.  Fractal modeling and segmentation for the enhancement of microcalcifications in digital mammograms , 1997, IEEE Transactions on Medical Imaging.

[30]  John D. Austin,et al.  Adaptive histogram equalization and its variations , 1987 .

[31]  Yianni Attikiouzel,et al.  Automatic Breast Border Segmentation by Background Modelling and Subtraction , 2000 .

[32]  Daniel Rueckert,et al.  Nonrigid registration using free-form deformations: application to breast MR images , 1999, IEEE Transactions on Medical Imaging.

[33]  L. Tabár,et al.  Potential contribution of computer-aided detection to the sensitivity of screening mammography. , 2000, Radiology.

[34]  Robert B. Jaffe.  Performance of Screening Mammography Among Women With and Without a First-Degree Relative With Breast Cancer , 2001 .

[35]  F. Winsberg,et al.  Detection of Radiographic Abnormalities in Mammograms by Means of Optical Scanning and Computer Analysis , 1967 .

[36]  E. R. Davies.  Algorithms for inspection: constraints, trade-offs and the design process , 1997 .

[37]  G W Seeley,et al.  Design and testing of artifact-suppressed adaptive histogram equalization: a contrast-enhancement technique for display of digital chest radiographs. , 1990, Journal of thoracic imaging.

[38]  M. Strintzis,et al.  Still Image Objective Segmentation Evaluation using Ground Truth , 2003 .

[39]  Guido Gerig,et al.  Valmet: A New Validation Tool for Assessing and Improving 3D Object Segmentation , 2001, MICCAI.

[40]  Stephen M. Pizer,et al.  Structure-sensitive adaptive contrast enhancement methods and their evaluation , 1993, Image Vis. Comput..

[41]  W. P. Evans,et al.  Re: Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. , 2004, Journal of the National Cancer Institute.

[42]  Malur K. Sundareshan,et al.  Adaptive image contrast enhancement based on human visual properties , 1994, IEEE Trans. Medical Imaging.

[43]  John C. Munson.  Software faults, software failures and software reliability modeling , 1996, Inf. Softw. Technol..

[44]  R M Rangayyan,et al.  Feature enhancement of film mammograms using fixed and adaptive neighborhoods. , 1984, Applied optics.

[45]  D. Louis Collins,et al.  Design and construction of a realistic digital brain phantom , 1998, IEEE Transactions on Medical Imaging.

[46]  Olli Nevalainen,et al.  Interactive Segmentation of the Breast Region from Digitized Mammograms with United Snakes , 1999 .

[47]  C. Boggis,et al.  Double reading of mammography screening films--one radiologist or two. , 1994, Clinical radiology.

[48]  Yongmin Kim,et al.  A methodology for evaluation of boundary detection algorithms on medical images , 1997, IEEE Transactions on Medical Imaging.

[49]  Max A. Viergever,et al.  A discrete dynamic contour model , 1995, IEEE Trans. Medical Imaging.

[50]  Michael A. Wirth,et al.  Segmentation of the breast region in mammograms using snakes , 2004, First Canadian Conference on Computer and Robot Vision, 2004. Proceedings..

[51]  D. Ikeda,et al.  Mammographic characteristics of 115 missed cancers later detected with screening mammography and the potential utility of computer-aided detection. , 2001, Radiology.

[52]  P. G. Tahoces,et al.  Automatic detection of breast border and nipple in digital mammograms. , 1996, Computer methods and programs in biomedicine.

[53]  M. Reiser,et al.  Computerassistierte Analyse von Mammographien in der klinischen Routinediagnostik , 1998, Der Radiologe.

[54]  Kunio Doi,et al.  Performance evaluation of an advanced method for automated identification of view positions of chest radiographs by use of a large database , 2002, SPIE Medical Imaging.

[55]  Karel J. Zuiderveld,et al.  Contrast Limited Adaptive Histogram Equalization , 1994, Graphics Gems.

[56]  R. Lucht,et al.  Comparison of rigid and elastic matching of dynamic magnetic resonance mammographic images by mutual information. , 2000, Medical physics.

[57]  Fátima L. S. Nunes,et al.  Performance of a processing scheme for clustered microcalcifications detection with different images database , 2000, Proceedings of the 22nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Cat. No.00CH37143).

[58]  Berkman Sahiner,et al.  Breast cancer detection: evaluation of a mass-detection algorithm for computer-aided diagnosis -- experience in 263 patients. , 2002, Radiology.

[59]  P. G. Tahoces,et al.  Enhancement of chest and breast radiographs by automatic spatial filtering. , 1991, IEEE transactions on medical imaging.

[60]  W. T. Ho,et al.  Clinical performance of computer-assisted detection (CAD system in detecting carcinoma in breasts of different densities. , 2003, Clinical radiology.

[61]  C J Vyborny,et al.  Can computers help radiologists read mammograms? , 1994, Radiology.

[62]  Impact of computer-placed prompts on sensitivity and specificity with different groups of mammographic film readers , 2003 .

[63]  D. Nikitenko,et al.  A fuzzy approach to segmenting the breast region in mammograms , 2004, IEEE Annual Meeting of the Fuzzy Information, 2004. Processing NAFIPS '04..

[64]  H. Sittek,et al.  Computer-aided diagnosis in mammography , 1997, Der Radiologe.

[65]  J F Urías.  Feature enhancement of images using maximal contrast pixel to pixel. , 1991, Applied optics.

[66]  Henry Petroski,et al.  To Engineer is Human , 1987 .

[67]  N. Petrick,et al.  Computer-aided classification of mammographic masses and normal tissue: linear discriminant analysis in texture feature space. , 1995, Physics in medicine and biology.

[68]  W A Kaiser,et al.  Reproducibility--an important factor determining the quality of computer-aided detection (CAD) systems. , 2000, European journal of radiology.

[69]  R M Nishikawa.  Mammographic databases. , 1998, Breast disease.

[70]  R. Chandrasekhar,et al.  Gross segmentation of mammograms using a polynomial model , 1996, Proceedings of 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[71]  R E Hendrick,et al.  Proposition: All mammograms should be double-read. , 1999, Medical physics.

[72]  Robi Polikar,et al.  Dynamic segmentation of breast tissue in digitized mammograms , 2001, 2001 Conference Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[73]  T. Freer,et al.  Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. , 2001, Radiology.

[74]  Patrick Courtney,et al.  Algorithmic Modeling for Performance Evaluation , 1997 .

[75]  Jian Fan,et al.  Mammographic feature enhancement by multiscale analysis , 1994, IEEE Trans. Medical Imaging.

[76]  Luisa P. Wallace,et al.  Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. , 2004, Journal of the National Cancer Institute.

[77]  Rangaraj M. Rangayyan,et al.  An indexed atlas of digital mammograms , 2003 .

[78]  Murk J. Bottema,et al.  Computer-aided screening mammography , 1999, Smart Materials, Nano-, and Micro- Smart Systems.

[79]  H P Chan,et al.  Automated detection of breast masses on mammograms using adaptive contrast enhancement and texture classification. , 1996, Medical physics.

[80]  Brian C. Lovell,et al.  Progress in the robust automated segmentation of real cell images , 1999, Smart Materials, Nano-, and Micro- Smart Systems.

[81]  N. Obuchowski,et al.  Comparing the performance of mammographic enhancement algorithms: a preference study. , 2000, AJR. American journal of roentgenology.

[82]  Rangaraj M. Rangayyan,et al.  Automatic identification of the pectoral muscle in mammograms , 2004, IEEE Transactions on Medical Imaging.

[83]  Guillermo H. Kaufmann,et al.  Contrast enhancement of mammographic features: a comparison of four methods , 1996 .

[84]  Thomas Boehm,et al.  Tumour detection rate of a new commercially available computer-aided detection system , 2001, European Radiology.

[85]  Andrew D. A. Maidment,et al.  Image processing algorithms for digital mammography: a pictorial essay. , 2000, Radiographics : a review publication of the Radiological Society of North America, Inc.

[86]  H L Kundel,et al.  Nature of expertise in searching mammograms for breast masses , 1996, Medical Imaging.

[87]  Ramachandran Chandrasekhar,et al.  Segmenting the Breast Border and Nipple on Mammograms , 2000 .

[88]  D. Dance,et al.  Automatic computer detection of clustered calcifications in digital mammograms , 1990, Physics in medicine and biology.

[89]  Bin Zheng,et al.  Mammography with computer-aided detection: reproducibility assessment initial experience. , 2003, Radiology.

[90]  Zhou Wang,et al.  Why is image quality assessment so difficult? , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[91]  M Kallergi,et al.  Evaluating the performance of detection algorithms in digital mammography. , 1999, Medical physics.

[92]  R. Bird,et al.  Analysis of cancers missed at screening mammography. , 1992, Radiology.

[93]  Ian W. Ricketts,et al.  The Mammographic Image Analysis Society digital mammogram database , 1994 .