How to Discriminate between Computer-Aided and Computer-Hindered Decisions

Background. Computer aids can affect decisions in complex ways, potentially even making them worse; common assessment methods may miss these effects. We developed a method for estimating the quality of decisions, as well as how computer aids affect it, and applied it to computer-aided detection (CAD) of cancer, reanalyzing data from a published study where 50 professionals (“readers”) interpreted 180 mammograms, both with and without computer support. Method. We used stepwise regression to estimate how CAD affected the probability of a reader making a correct screening decision on a patient with cancer (sensitivity), thereby taking into account the effects of the difficulty of the cancer (proportion of readers who missed it) and the reader’s discriminating ability (Youden’s determinant). Using regression estimates, we obtained thresholds for classifying a posteriori the cases (by difficulty) and the readers (by discriminating ability). Results. Use of CAD was associated with a 0.016 increase in sensitivity (95% confidence interval [CI], 0.003–0.028) for the 44 least discriminating radiologists for 45 relatively easy, mostly CAD-detected cancers. However, for the 6 most discriminating radiologists, with CAD, sensitivity decreased by 0.145 (95% CI, 0.034–0.257) for the 15 relatively difficult cancers. Conclusions. Our exploratory analysis method reveals unexpected effects. It indicates that, despite the original study detecting no significant average effect, CAD helped the less discriminating readers but hindered the more discriminating readers. Such differential effects, although subtle, may be clinically significant and important for improving both computer algorithms and protocols for their use. They should be assessed when evaluating CAD and similar warning systems.

[1]  L. Tabár,et al.  Potential contribution of computer-aided detection to the sensitivity of screening mammography. , 2000, Radiology.

[2]  Paul Taylor,et al.  Computer aids and human second reading as interventions in screening mammography: two systematic reviews to compare effects on cancer detection and recall rate. , 2008, European journal of cancer.

[3]  J. Sterne Essentials of Medical Statistics , 1991 .

[4]  石黒 真木夫,et al.  Akaike information criterion statistics , 1986 .

[5]  G. Kitagawa,et al.  Akaike Information Criterion Statistics , 1988 .

[6]  R. F. Wagner,et al.  Reader Variability in Mammography and Its Implications for Expected Utility over the Population of Readers and Cases , 2004, Medical decision making : an international journal of the Society for Medical Decision Making.

[7]  Lorenzo Strigini,et al.  Human-machine diversity in the use of computerised advisory systems: a case study , 2003, 2003 International Conference on Dependable Systems and Networks, 2003. Proceedings..

[8]  J. C. van Houwelingen,et al.  A goodness-of-fit test for binary regression models, based on smoothing methods , 1991 .

[9]  Abdul V. Roudsari,et al.  Automation bias: a systematic review of frequency, effect mediators, and mitigators , 2012, J. Am. Medical Informatics Assoc..

[10]  Lorenzo Strigini,et al.  Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography. , 2004, Academic radiology.

[11]  Bev Littlewood,et al.  Modeling software design diversity: a review , 2001, CSUR.

[12]  W. Youden,et al.  Index for rating diagnostic tests , 1950, Cancer.

[13]  P. Taylor,et al.  Impact of computer-aided detection prompts on the sensitivity and specificity of screening mammography. , 2005, Health technology assessment.

[14]  Wendy Bruening,et al.  Computer-aided detection mammography for breast cancer screening: systematic review and meta-analysis , 2009, Archives of Gynecology and Obstetrics.

[15]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .

[16]  Jason A. Lyman,et al.  Clinical decision support: progress and opportunities , 2010, J. Am. Medical Informatics Assoc..

[17]  Kathleen L. Mosier,et al.  Accountability and automation bias , 2000, Int. J. Hum. Comput. Stud..

[18]  Eric R. Ziegel,et al.  An Introduction to Generalized Linear Models , 2002, Technometrics.

[19]  Bev Littlewood,et al.  Conceptual Modeling of Coincident Failures in Multiversion Software , 1989, IEEE Trans. Software Eng..

[20]  P. Taylor,et al.  An evaluation of the impact of computer-based prompts on screen readers' interpretation of mammograms. , 2004, The British journal of radiology.

[21]  F. J. Anscombe,et al.  On estimating binomial response relations , 1956 .

[22]  D. Hosmer,et al.  A comparison of goodness-of-fit tests for the logistic regression model. , 1997, Statistics in medicine.

[23]  B. Zheng,et al.  Soft-copy mammographic readings with different computer-assisted detection cuing environments: preliminary findings. , 2001, Radiology.

[24]  Eugenio Alberdi,et al.  Effects of incorrect CAD output on human decision making in mammography , 2004 .

[25]  S. Astley,et al.  Single reading with computer-aided detection for screening mammography. , 2008, The New England journal of medicine.

[26]  Giovanni Luca Christian Masala,et al.  Computer Aided Detection on Mammography , 2008 .

[27]  P. Taylor,et al.  Assessing the impact of CAD on the sensitivity and specificity of film readers. , 2004, Clinical radiology.

[28]  N. Bingham,et al.  Generalised Linear Models , 2010 .

[29]  Luisa P. Wallace,et al.  Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. , 2004, Journal of the National Cancer Institute.

[30]  P. Ayton,et al.  Use of computer-aided detection (CAD) tools in screening mammography: a multidisciplinary investigation. , 2005, The British journal of radiology.

[31]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[32]  Lorenzo Strigini,et al.  Why Are People's Decisions Sometimes Worse with Computer Support? , 2009, SAFECOMP.