The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans.

PURPOSE The development of computer-aided diagnostic (CAD) methods for lung nodule detection, classification, and quantitative assessment can be facilitated through a well-characterized repository of computed tomography (CT) scans. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) completed such a database, establishing a publicly available reference for the medical imaging research community. Initiated by the National Cancer Institute (NCI), further advanced by the Foundation for the National Institutes of Health (FNIH), and accompanied by the Food and Drug Administration (FDA) through active participation, this public-private partnership demonstrates the success of a consortium founded on a consensus-based process. METHODS Seven academic centers and eight medical imaging companies collaborated to identify, address, and resolve challenging organizational, technical, and clinical issues to provide a solid foundation for a robust database. The LIDC/IDRI Database contains 1018 cases, each of which includes images from a clinical thoracic CT scan and an associated XML file that records the results of a two-phase image annotation process performed by four experienced thoracic radiologists. In the initial blinded-read phase, each radiologist independently reviewed each CT scan and marked lesions belonging to one of three categories ("nodule > or =3 mm," "nodule <3 mm," and "non-nodule > or =3 mm"). In the subsequent unblinded-read phase, each radiologist independently reviewed their own marks along with the anonymized marks of the three other radiologists to render a final opinion. The goal of this process was to identify as completely as possible all lung nodules in each CT scan without requiring forced consensus. RESULTS The Database contains 7371 lesions marked "nodule" by at least one radiologist. 2669 of these lesions were marked "nodule > or =3 mm" by at least one radiologist, of which 928 (34.7%) received such marks from all four radiologists. These 2669 lesions include nodule outlines and subjective nodule characteristic ratings. CONCLUSIONS The LIDC/IDRI Database is expected to provide an essential medical imaging research resource to spur CAD development, validation, and dissemination in clinical practice.

Richard C. Pais | Rachael Y. Roberts | Gary E Laderach | Ali O. Farooqi | E. Hoffman | G. McLennan | S. Armato | B. Sundaram | M. Salganicoff | Lori E. Dodd | L. Schwartz | Binsheng Zhao | L. Clarke | E. Kazerooni | D. Gur | A. Reeves | L. Bidaut | M. McNitt-Gray | C. Meyer | D. Aberle | C. Henschke | H. MacMahon | Edwin J R Van Beeke | D. Yankelevitz | A. Biancardi | P. Bland | Matthew S. Brown | R. Engelmann | G. Laderach | D. Max | D. Qing | R. Roberts | A. R. Smith | A. Starkey | Poonam Batrah | P. Caligiuri | G. Gladish | C. Jude | R. Munden | I. Petkovska | L. Quint | L. Dodd | C. Fenimore | N. Petrick | J. Freymann | J. Kirby | Brian Hughes | Alessi Vande Casteele | S. Gupte | Maha Sallamm | Michael Heath | M. Kuhn | E. Dharaiya | Richard Burns | David Fryd | V. Anand | U. Shreter | S. Vastagh | B. Croft | E. V. Beek | M. Sallam | Anthony P. Reeves | M. Brown | Poonam Batra | Daniel Max | Matthew S. Brown | Amanda R. Smith | A. R. Smith | Amanda R. Smith | E. Hoffman | Binsheng Zhao | Matthew S. Brown | Gary E. Laderach | David P.-y. Qing | Amanda R. Smith | Brian Hughes | Alessi Vande Casteele | Michael Kuhn | Binsheng Zhao | Matthew S. Brown | Edwin J R Van Beeke | Poonam Batrah | Maha Sallamm | L E Quint | Poonam Batra | Eric A. Hoffman | Geoffrey Mclennan | Laurence P. Clarke | Ella A. Kazerooni | Lawrence H. Schwartz | Nicholas Petrick | Luc Bidaut | C. R. Meyer | Denise Aberle | Eric A. Hoffman | Edwin J R van Beek | David Yankelevitz | Alberto Biancardi | Matthew S. Brown | Daniel Max | David P.-Y. Qing | Amanda R. Smith | C. Matilda Jude | Lawrence H. Schwartz | Lori E. Dodd | David Gur | Justin Kirby | Brian Hughes | Alessi Vande Casteele | Sangeeta Gupte | Maha Sallam | Michael D. Heath | Michael H. Kuhn | Richard Burns | Uri Shreter | Barbara Y. Croft | Binsheng Zhao | Denise R Aberle | U. Shreter | Denise R. Aberle | Alberto M. Biancardi | Adam Starkey

[1]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[2]  Robert M. Nishikawa.  Design of a common database for research in mammogram image analysis , 1993, Electronic Imaging.

[3]  K Doi,et al.  Effect of case selection on the performance of computer-aided detection schemes. , 1994, Medical physics.

[4]  J. Austin,et al.  Glossary of terms for CT of the lungs: recommendations of the Nomenclature Committee of the Fleischner Society. , 1996, Radiology.

[5]  L P Clarke,et al.  Medical image databases for CAD applications in digital mammography: design issues. , 1997, Studies in health technology and informatics.

[6]  Richard H. Moore,et al.  Current Status of the Digital Database for Screening Mammography , 1998, Digital Mammography / IWDM.

[7]  Robert M. Nishikawa,et al.  Variations in measured performance of CAD schemes due to database composition and scoring protocol , 1998, Medical Imaging.

[8]  K. Doi,et al.  Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists' detection of pulmonary nodules. , 2000, AJR. American journal of roentgenology.

[9]  L. Clarke,et al.  National Cancer Institute initiative: Lung image database resource for imaging research. , 2001, Academic radiology.

[10]  E. Hoffman,et al.  Lung image database consortium: developing a resource for the medical imaging research community. , 2004, Radiology.

[11]  R. F. Wagner,et al.  Assessment methodologies and statistical issues for computer-aided diagnosis of lung nodules in computed tomography: contemporary research topics relevant to the lung image database consortium. , 2004, Academic radiology.

[12]  E. Paulson,et al.  Analysis of air contrast barium enema, computed tomographic colonography, and colonoscopy: prospective comparison , 2005, The Lancet.

[13]  Richard C. Pais,et al.  Evaluation of Lung MDCT Nodule Annotation Across Radiologists and Methods 1 , 2006 .

[14]  Mathias Prokop,et al.  Pulmonary nodules detected at lung cancer screening: interobserver variability of semiautomated volume measurements. , 2006, Radiology.

[15]  D. Xu,et al.  Nodule management protocol of the NELSON randomised lung cancer screening trial. , 2006, Lung cancer.

[16]  Stephen M. Moore,et al.  Creation of a CT Image Library for the Lung Screening Study of the National Lung Screening Trial , 2007, Journal of Digital Imaging.

[17]  E. V. van Beek,et al.  The Lung Image Database Consortium (LIDC): a comparison of different size metrics for pulmonary nodule measurements. , 2007, Academic radiology.

[18]  J. Habbema,et al.  Risk‐based selection from the general population in a screening trial: Selection criteria, recruitment and power for the Dutch‐Belgian randomised lung cancer multi‐slice CT screening trial (NELSON) , 2007, International journal of cancer.

[19]  Michael F. McNitt-Gray,et al.  Forming a reference standard from LIDC data: impact of reader agreement on reported CAD performance , 2007, SPIE Medical Imaging.

[20]  Heber MacMahon,et al.  The Lung Image Database Consortium (LIDC): ensuring the integrity of expert-defined "truth". , 2007, Academic radiology.

[21]  Iva Petkovska,et al.  The effect of lung volume on nodule size on CT. , 2007, Academic radiology.

[22]  Zaid J. Towfic,et al.  The Lung Image Database Consortium (LIDC) data collection process for nodule detection and annotation , 2007, SPIE Medical Imaging.

[23]  Kenji Suzuki,et al.  CT colonography: false-negative interpretations. , 2007, Radiology.

[24]  Geoffrey McLennan,et al.  The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans. , 2007, Academic radiology.

[25]  Richard H. Moore,et al.  THE DIGITAL DATABASE FOR SCREENING MAMMOGRAPHY , 2007 .

[26]  Harry J de Koning,et al.  Limited value of shape, margin and CT density in the discrimination between benign and malignant screen detected solid pulmonary nodules of the NELSON trial. , 2008, European journal of radiology.

[27]  S. Armato,et al.  Discrete-space versus continuous-space lesion boundary and area definitions. , 2008, Medical physics.

[28]  M. Giger,et al.  Anniversary paper: History and status of CAD and quantitative image analysis: the role of Medical Physics and AAPM. , 2008, Medical physics.

[29]  L P Clarke,et al.  The Reference Image Database to Evaluate Response to Therapy in Lung Cancer (RIDER) Project: A Resource for the Development of Change‐Analysis Software , 2008, Clinical pharmacology and therapeutics.

[30]  Anthony P. Reeves,et al.  A comparison of ground truth estimation methods , 2010, International Journal of Computer Assisted Radiology and Surgery.

[31]  Bram van Ginneken,et al.  A large-scale evaluation of automatic pulmonary nodule detection in chest CT using local image features and k-nearest-neighbour classification , 2009, Medical Image Anal..

[32]  A. P. Reeves,et al.  A public image database to support research in computer aided diagnosis , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[33]  Nicholas Petrick,et al.  Quantitative imaging to assess tumor response to therapy: common themes of measurement, truth data, and error sources. , 2009, Translational oncology.

[34]  Harry J de Koning,et al.  Role of baseline nodule density and changes in density and nodule features in the discrimination between benign and malignant solid indeterminate pulmonary nodules. , 2009, European journal of radiology.

[35]  Jacob D. Furst,et al.  Predicting Radiological Panel Opinions Using a Panel of Machine Learning Classifiers , 2009, Algorithms.

[36]  Jacob D. Furst,et al.  Evaluation Challenges for Bridging Semantic Gap: Shape Disagreements on Pulmonary Nodules in the Lung Image Database Consortium , 2009, Int. J. Heal. Inf. Syst. Informatics.

[37]  B. van Ginneken,et al.  Automatic lung segmentation from thoracic computed tomography scans using a hybrid approach with error detection. , 2009, Medical physics.

[38]  Maria C. Carrillo,et al.  Maximizing the Alzheimer's Disease Neuroimaging Initiative II , 2009, Alzheimer's & Dementia.

[39]  Daniel L. Rubin,et al.  A Semantic Image Annotation Model to Enable Integrative Translational Research , 2009, Summit on translational bioinformatics.

[40]  Geoffrey McLennan,et al.  Assessment of radiologist performance in the detection of lung nodules: dependence on the definition of "truth". , 2009, Academic radiology.

[41]  Ilaria Gori,et al.  Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: The ANODE09 study , 2010, Medical Image Anal..

[42]  Nicholas Petrick,et al.  Volumetric CT in lung cancer: an example for the qualification of imaging as a biomarker. , 2010, Academic radiology.

[43]  J Anthony Seibert,et al.  Normalized CT dose index of the CT scanners used in the National Lung Screening Trial. , 2010, AJR. American journal of roentgenology.