Heat maps: an aid for data analysis and understanding of ROC CAD experiments.

RATIONALE AND OBJECTIVES To demonstrate the value of a new data visualization and exploration method for mutlireader-multicase receiver operating characteristic (MRMC-ROC) experiments of computer-aided detection (CAD) algorithms that uses three-dimensional (3D) heat maps tool adapted from gene expression array analysis. MATERIALS AND METHODS We are using data from a clinical trial of a commercial CAD system for lung cancer detection (RapidScreen RS-2000, Riverain Medical Group, Miamisburg, OH, and Rockville, MD). 3D heat maps, originally developed for displaying changes in gene expression after cancer chemotherapy in MATLAB, were modified to display the radiologists confidence levels as they interpreted chest radiographs and used to visualize the radiologists confidence levels before and after the provision of a CAD system. RESULTS Heat maps demonstrated the variation among radiologists in their interpretation, and the degree of variation in interpretation when a single radiologist reinterpreted the same case without and with CAD modality. They demonstrated the variability in the identification of each cancer/cancer-free case and the variability of change seen when CAD prompts were provided. CONCLUSIONS CAD increases the consistency of interpretation of a single radiologist and of a group of radiologists. Heat maps provide a method for data visualization that clarifies the effects of reader variability in ROC CAD experiments. We demonstrated how heat maps can be used to document the complexity of reader variability and suggested how clustering can reveal both nonintuitive and intuitive groupings of cases, readers, and the interaction of both with CAD.

[1]  K. Doi,et al.  Potential of computer-aided diagnosis to reduce variability in radiologists' interpretations of mammograms depicting microcalcifications. , 2001, Radiology.

[2]  M. Tockman,et al.  Radiographic screening in the early detection of lung cancer. , 1978, Radiologic clinics of North America.

[3]  D. Botstein,et al.  Cluster analysis and display of genome-wide expression patterns. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[4]  J. Austin,et al.  Missed bronchogenic carcinoma: radiographic findings in 27 patients with a potentially resectable lesion evident in retrospect. , 1992, Radiology.

[5]  J. Yerushalmy.  The statistical assessment of the variability in observer perception and description of roentgenographic pulmonary shadows. , 1969, Radiologic clinics of North America.

[6]  J. Elmore,et al.  Screening mammograms by community radiologists: variability in false-positive rates. , 2002, Journal of the National Cancer Institute.

[7]  K. Doi,et al.  Effect of a computer-aided diagnosis scheme on radiologists' performance in detection of lung nodules on radiographs. , 1996, Radiology.

[8]  Eun Sil Kim,et al.  Chest radiology. , 1992, Current opinion in radiology.

[9]  Robert M. Nishikawa,et al.  Independent versus sequential reading in ROC studies of computer-assist modalities , 2002, SPIE Medical Imaging.

[10]  Teresa Osicka,et al.  Reader variability: what we can learn from computer-aided detection experiments. , 2006, Journal of the American College of Radiology : JACR.

[11]  Emily F Conant,et al.  Association of volume and volume-independent factors with accuracy in screening mammogram interpretation. , 2003, Journal of the National Cancer Institute.

[12]  C. D'Orsi,et al.  Accuracy of screening mammography interpretation by characteristics of radiologists. , 2004, Journal of the National Cancer Institute.

[13]  Tommi S. Jaakkola,et al.  Fast optimal leaf ordering for hierarchical clustering , 2001, ISMB.

[14]  Kunio Doi,et al.  Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance. , 2002, Academic radiology.

[15]  Stitik Fp,et al.  Radiographic screening in the early detection of lung cancer. , 1978, Radiologic clinics of North America.

[16]  K. Awai,et al.  Pulmonary nodules at chest CT: effect of computer-aided diagnosis on radiologists' detection performance. , 2004, Radiology.

[17]  J. V. van Engelshoven,et al.  Miss rate of lung cancer on the chest radiograph in clinical practice. , 1999, Chest.

[18]  J. Austin,et al.  Missed non-small cell lung cancer: radiographic findings of potentially resectable lesions evident only in retrospect. , 2003, Radiology.