Standardized evaluation framework for evaluating coronary artery stenosis detection, stenosis quantification and lumen segmentation algorithms in computed tomography angiography

Though conventional coronary angiography (CCA) has been the standard of reference for diagnosing coronary artery disease in the past decades, computed tomography angiography (CTA) has rapidly emerged, and is nowadays widely used in clinical practice. Here, we introduce a standardized evaluation framework to reliably evaluate and compare the performance of the algorithms devised to detect and quantify the coronary artery stenoses, and to segment the coronary artery lumen in CTA data. The objective of this evaluation framework is to demonstrate the feasibility of dedicated algorithms to: (1) (semi-)automatically detect and quantify stenosis on CTA, in comparison with quantitative coronary angiography (QCA) and CTA consensus reading, and (2) (semi-)automatically segment the coronary lumen on CTA, in comparison with expert's manual annotation. A database consisting of 48 multicenter multivendor cardiac CTA datasets with corresponding reference standards are described and made available. The algorithms from 11 research groups were quantitatively evaluated and compared. The results show that (1) some of the current stenosis detection/quantification algorithms may be used for triage or as a second-reader in clinical practice, and that (2) automatic lumen segmentation is possible with a precision similar to that obtained by experts. The framework is open for new submissions through the website, at http://coronary.bigr.nl/stenoses/.

[1]  Jonathan G Goldin,et al.  Assessment of Coronary Artery Disease by Cardiac Computed Tomography: A Scientific Statement From the American Heart Association Committee on Cardiovascular Imaging and Intervention, Council on Cardiovascular Radiology and Intervention, and Committee on Cardiac Imaging, Council on Clinical Cardiolog , 2006, Circulation.

[2]  Jouke Dijkstra,et al.  Automated quantification of stenosis severity on 64-slice CT: a comparison with quantitative coronary angiography. , 2010, JACC. Cardiovascular imaging.

[3]  Rozemarijn Vliegenthart,et al.  Diagnostic performance of coronary CT angiography for stenosis detection according to calcium score: systematic review and meta-analysis , 2012, European Radiology.

[4]  A R Benedetto,et al.  Accurate estimates of absolute left ventricular volumes from equilibrium radionuclide angiographic count data using a simple geometric attenuation correction. , 1984, Journal of the American College of Cardiology.

[5]  Renu Virmani,et al.  Pathology of the vulnerable plaque. , 2007, Journal of the American College of Cardiology.

[6]  C J Slager,et al.  Experimental validation of geometric and densitometric coronary measurements on the new generation Cardiovascular Angiography Analysis System (CAAS II). , 1993, Catheterization and cardiovascular diagnosis.

[7]  Stephan Achenbach,et al.  Can CT detect the vulnerable coronary plaque? , 2008, The International Journal of Cardiovascular Imaging.

[8]  Robert Califf,et al.  Value of the History and Physical in Identifying Patients at Increased Risk for Coronary Artery Disease , 1993, Annals of Internal Medicine.

[9]  Nicholas Ayache,et al.  Medical Image Analysis: Progress over Two Decades and the Challenges Ahead , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  R. Frye,et al.  A reporting system on patients evaluated for coronary artery disease. Report of the Ad Hoc Committee for Grading of Coronary Artery Disease, Council on Cardiovascular Surgery, American Heart Association. , 1975, Circulation.

[11]  A. Weustink,et al.  The role of multi-slice computed tomography in stable angina management: a current perspective , 2011, Netherlands heart journal : monthly journal of the Netherlands Society of Cardiology and the Netherlands Heart Foundation.

[12]  Theo van Walsum,et al.  Automatic Stenoses Detection , Quantification and Lumen Segmentation of the Coronary Arteries using a Two Point Centerline Extraction Scheme , 2012 .

[13]  Stefan Wesarg,et al.  Localizing Calcifications in Cardiac CT Data Sets Using a New Vessel Segmentation Approach , 2006, Journal of Digital Imaging.

[14]  Günther Greiner,et al.  Multi-scale feature extraction for learning-based classification of coronary artery stenosis , 2009, Medical Imaging.

[15]  Hüseyin Tek,et al.  Robust Vessel Tree Modeling , 2008, MICCAI.

[16]  G. Unal,et al.  Automatic detection of coronary artery stenosis in CTA based on vessel intensity and geometric features , 2012 .

[17]  Dorin Comaniciu,et al.  Fast Automatic Detection of Calcified Coronary Lesions in 3D Cardiac CT Images , 2010, MLMI.

[18]  Morton J. Kern,et al.  Why all the focus on cardiac imaging? , 2010, JACC. Cardiovascular imaging.

[19]  Berkman Sahiner,et al.  Automated segmentation and tracking of coronary arteries in cardiac CT scans: comparison of performance with a clinically used commercial software , 2010, Medical Imaging.

[20]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[21]  Martin Styner,et al.  Standardized evaluation methodology and reference database for evaluating coronary artery centerline extraction algorithms , 2009, Medical Image Anal..

[22]  Eric Boersma,et al.  Comparison of the value of coronary calcium detection to computed tomographic angiography and exercise testing in patients with chest pain. , 2009, The American journal of cardiology.

[23]  Stefan Wesarg,et al.  Facilitating coronary artery evaluation in MDCT using a 3D automatic vessel segmentation tool , 2006, European Radiology.

[24]  Maria A. Zuluaga,et al.  Automatic detection of abnormal vascular cross-sections based on density level detection and support vector machines , 2010, International Journal of Computer Assisted Radiology and Surgery.

[25]  T van Walsum,et al.  Coronary centerline extraction from CT coronary angiography images using a minimum cost path approach. , 2009, Medical physics.

[26]  I. Magnin,et al.  Fast-marching contours for the segmentation of vessel lumen in CTA cross-sections , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[27]  Daniel S. Berman,et al.  MULTICENTER EVALUATION OF DUAL SOURCE CT CORONARY ANGIOGRAPHY IN PATIENTS WITH INTERMEDIATE LIKELIHOOD OF CORONARY ARTERY STENOSES (MEDIC): ACCURACY FOR THE DETECTION OF INDIVIDUALS WITH SIGNIFICANT CORONARY ARTERY STENOSES , 2012 .

[28]  Leo Joskowicz,et al.  Evaluation framework for carotid bifurcation lumen segmentation and stenosis grading , 2011, Medical Image Anal..

[29]  E. Halpern,et al.  Diagnosis of coronary stenosis with CT angiography comparison of automated computer diagnosis with expert readings. , 2011, Academic radiology.

[30]  Marcela Hernández Hoyos,et al.  Coronary artery segmentation and stenosis quantification in CT images with use of a right generalized cylinder model , 2012 .

[31]  Mathias Prokop,et al.  Diagnostic accuracy of 64-slice computed tomography coronary angiography: a prospective, multicenter, multivendor study. , 2008, Journal of the American College of Cardiology.

[32]  Anthony J. Yezzi,et al.  Vessel Tractography Using an Intensity Based Tensor Model With Branch Detection , 2013, IEEE Transactions on Medical Imaging.

[33]  Jeroen J. Bax,et al.  Automatic centerline extraction of coronary arteries in coronary computed tomographic angiography , 2011, The International Journal of Cardiovascular Imaging.

[34]  Roman Goldenberg,et al.  Computer-aided simple triage (CAST) for coronary CT angiography (CCTA) , 2012, International Journal of Computer Assisted Radiology and Surgery.

[35]  Theo van Walsum,et al.  Coronary Lumen Segmentation Using Graph Cuts and Robust Kernel Regression , 2009, IPMI.

[36]  Alejandro F. Frangi,et al.  Muliscale Vessel Enhancement Filtering , 1998, MICCAI.

[37]  James S. Duncan,et al.  Medical Image Analysis , 1999, IEEE Pulse.

[38]  Jouke Dijkstra,et al.  Automated quantification of coronary plaque with computed tomography: comparison with intravascular ultrasound using a dedicated registration algorithm for fusion-based quantification. , 2012, European heart journal.

[39]  William M. Wells,et al.  Medical Image Computing and Computer-Assisted Intervention — MICCAI’98 , 1998, Lecture Notes in Computer Science.

[40]  Roman Goldenberg,et al.  Computer-aided simple triage , 2011, International Journal of Computer Assisted Radiology and Surgery.

[41]  Max A. Viergever,et al.  On Combining Computer-Aided Detection Systems , 2011, IEEE Transactions on Medical Imaging.

[42]  Jeroen J. Bax,et al.  Automatic coronary artery tree labeling in coronary computed tomographic angiography datasets , 2011, 2011 Computing in Cardiology.

[43]  Isabelle Bloch,et al.  A review of 3D vessel lumen segmentation techniques: Models, features and extraction schemes , 2009, Medical Image Anal..

[44]  D. Berman,et al.  SCCT guidelines for the interpretation and reporting of coronary computed tomographic angiography. , 2009, Journal of cardiovascular computed tomography.

[45]  D. Mozaffarian,et al.  Heart disease and stroke statistics--2012 update: a report from the American Heart Association. , 2012, Circulation.

[46]  Roman Goldenberg,et al.  Automated computer-aided stenosis detection at coronary CT angiography: initial experience , 2009, European Radiology.

[47]  Dorin Comaniciu,et al.  Detection, Grading and Classification of Coronary Stenoses in Computed Tomography Angiography , 2011, MICCAI.

[48]  T. Schacker,et al.  Clinical and Epidemiologic Features of Primary HIV Infection , 1996, Annals of Internal Medicine.

[49]  Yongyi Yang,et al.  Machine Learning in Medical Imaging , 2010, IEEE Signal Processing Magazine.

[50]  V. Fuster,et al.  Coronary plaque disruption. , 1995, Circulation.

[51]  Örjan Smedby,et al.  Vessel Segmentation Using Implicit Model-Guided Level Sets , 2012, MICCAI 2012.

[52]  Yan Yang,et al.  Quantification of coronary arterial stenoses in CTA using fuzzy distance transform , 2012, Comput. Medical Imaging Graph..

[53]  Jouke Dijkstra,et al.  FrenchCoast: Fast, Robust Extraction for the Nice CHallenge on COronary Artery Segmentation of the Tree , 2012 .

[54]  Ulrich Dietz,et al.  Comparison of QCA systems , 1997, The International Journal of Cardiac Imaging.

[55]  Armin Kanitsar,et al.  Shape and Appearance Models for Automatic Coronary Artery Tracking , 2008, The MIDAS Journal.

[56]  Philippe C. Cattin,et al.  Automatic Detection of Calcified Coronary Plaques in Computed Tomography Data Sets , 2008, MICCAI.