Deep learning can be used to train naïve, nonprofessional observers to detect diagnostic visual patterns of certain cancers in mammograms: a proof-of-principle study

Abstract. The scientific, clinical, and pedagogical significance of devising methodologies to train nonprofessional subjects to recognize diagnostic visual patterns in medical images has been broadly recognized. However, systematic approaches to doing so remain poorly established. Using mammography as an exemplar case, we use a series of experiments to demonstrate that deep learning (DL) techniques can, in principle, be used to train naïve subjects to reliably detect certain diagnostic visual patterns of cancer in medical images. In the main experiment, subjects were required to learn to detect statistical visual patterns diagnostic of cancer in mammograms using only the mammograms and feedback provided following the subjects’ response. We found not only that the subjects learned to perform the task at statistically significant levels, but also that their eye movements related to image scrutiny changed in a learning-dependent fashion. Two additional, smaller exploratory experiments suggested that allowing subjects to re-examine the mammogram in light of various items of diagnostic information may help further improve DL of the diagnostic patterns. Finally, a fourth small, exploratory experiment suggested that the image information learned was similar across subjects. Together, these results prove the principle that DL methodologies can be used to train nonprofessional subjects to reliably perform those aspects of medical image perception tasks that depend on visual pattern recognition expertise.

[1]  William Hendee,et al.  The Handbook of Medical Image Perception and Techniques. , 2010, Medical physics.

[2]  Sejong Yoon,et al.  AdaBoost-based multiple SVM-RFE for classification of mammograms in DDSM , 2008, 2008 IEEE International Conference on Bioinformatics and Biomeidcine Workshops.

[3]  Georgia D. Tourassi,et al.  Identifying error-making patterns in assessment of mammographic BI-RADS descriptors among radiology residents using statistical pattern recognition. , 2012, Academic radiology.

[4]  Jeremy M Wolfe.  Explicit expectations and the effects of prevalence. , 2011, Radiology.

[5]  Jeremy M Wolfe,et al.  When do I quit? The search termination problem in visual search. , 2012, Nebraska Symposium on Motivation. Nebraska Symposium on Motivation.

[6]  L. Nyström.  Screening mammography re-evaluated , 2000, The Lancet.

[7]  S. Martinez-Conde,et al.  Microsaccade Characteristics in Neurological and Ophthalmic Disease , 2018, Front. Neurol..

[8]  Elizabeth A. Krupinski.  Medical Imaging 2000: Image Perception and Performance , 2000 .

[9]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[10]  Ashok Kumar,et al.  Advances in Computational Science, Engineering and Information Technology , 2013, ICCS 2013.

[11]  Ziba Gandomkar,et al.  Radiologists can detect the ‘gist’ of breast cancer before any overt signs of cancer appear , 2018, Scientific Reports.

[12]  Lars J. Grimm,et al.  Radiology resident mammography training: interpretation difficulty and error-making patterns. , 2014, Academic radiology.

[13]  M. Kallergi.  Computer-aided diagnosis of mammographic microcalcification clusters. , 2004, Medical physics.

[14]  David Gur,et al.  "Memory effect" in observer performance studies of mammograms. , 2005, Academic radiology.

[15]  Elizabeth A. Krupinski,et al.  Pigeons (Columba livia) as Trainable Observers of Pathology and Radiology Breast Cancer Images , 2015, PloS one.

[16]  Harold L Kundel,et al.  History of research in medical image perception. , 2006, Journal of the American College of Radiology : JACR.

[17]  Carol A. Seger,et al.  Implicit learning. , 1994, Psychological bulletin.

[18]  M. Benndorf,et al.  Provision of the DDSM mammography metadata in an accessible format. , 2014, Medical physics.

[19]  J. Hegdé,et al.  Fragment-Based Learning of Visual Object Categories , 2008, Current Biology.

[20]  M. Helvie,et al.  Mammographic appearance of cancer in the opposite breast: comparison with the first cancer. , 1996, AJR. American journal of roentgenology.

[21]  Morten H. Christiansen,et al.  Domain generality versus modality specificity: the paradox of statistical learning , 2015, Trends in Cognitive Sciences.

[22]  H L Kundel,et al.  Interpreting chest radiographs without visual search. , 1975, Radiology.

[23]  Richard N Aslin,et al.  Statistical learning of new visual feature combinations by infants , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[24]  R. Shepard,et al.  How a cognitive psychologist came to seek universal laws , 2004, Psychonomic bulletin & review.

[25]  Theodore Y Kim,et al.  Lack of uniform diagnostic criteria for inflammatory breast cancer limits interpretation of treatment outcomes: a systematic review. , 2006, Clinical breast cancer.

[26]  Alexander C. Schütz,et al.  Eye movements and perception: a selective review. , 2011, Journal of vision.

[27]  M. Ibbotson,et al.  Visual perception and saccadic eye movements , 2011, Current Opinion in Neurobiology.

[28]  E. Conant,et al.  How experience and training influence mammography expertise. , 1999, Academic radiology.

[29]  R. Nosofsky.  Similarity Scaling and Cognitive Process Models , 1992 .

[30]  Fiona Fidler,et al.  STATISTICAL COGNITION: TOWARDS EVIDENCE-BASED PRACTICE IN STATISTICS AND STATISTICS EDUCATION , 2008 .

[31]  Nikolaos Doulamis,et al.  Deep Learning for Computer Vision: A Brief Review , 2018, Comput. Intell. Neurosci..

[32]  Xiaohong Zhang,et al.  Diagnostic Value of Nineteen Different Imaging Methods for Patients with Breast Cancer: a Network Meta-Analysis , 2018, Cellular Physiology and Biochemistry.

[33]  Edward E. Smith,et al.  The case for implicit category learning , 2008, Cognitive, affective & behavioral neuroscience.

[34]  David Dagan Feng,et al.  Content-Based Medical Image Retrieval: A Survey of Applications to Multidimensional and Multimodality Data , 2013, Journal of Digital Imaging.

[35]  Alima Damak Masmoudi,et al.  An efficient microcalcifications detection based on dual spatial/spectral processing , 2017, Multimedia Tools and Applications.

[36]  A. Vadivel,et al.  Classifying Digital Mammogram Masses Using Univariate ANOVA Discriminant Analysis , 2009, 2009 International Conference on Advances in Recent Technologies in Communication and Computing.

[37]  R. Aslin,et al.  Statistical learning of higher-order temporal structure from visual shape sequences. , 2002, Journal of experimental psychology. Learning, memory, and cognition.

[38]  Heather Carnahan,et al.  Integration of basic sciences and clinical sciences in oral radiology education for dental students. , 2013, Journal of dental education.

[39]  Eun-Kyung Kim,et al.  Application of Computer‐Aided Diagnosis on Breast Ultrasonography: Evaluation of Diagnostic Performances and Agreement of Radiologists According to Different Levels of Experience , 2018, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[40]  C. Langlotz,et al.  Deep Learning to Classify Radiology Free-Text Reports. , 2017, Radiology.

[41]  Jean-Marc Petit,et al.  Discovery Science , 2012, Lecture Notes in Computer Science.

[42]  Trafton Drew,et al.  Informatics in radiology: what can you see in a single glance and how might this guide visual search in medical images? , 2013, Radiographics : a review publication of the Radiological Society of North America, Inc.

[43]  J. Wolfe,et al.  From Perception to Consciousness: Searching with Anne Treisman , 2012 .

[44]  W. Woodward,et al.  Inflammatory breast cancer: what we know and what we need to learn. , 2012, The oncologist.

[45]  Hiba Chougrad,et al.  Deep Convolutional Neural Networks for breast cancer screening , 2018, Comput. Methods Programs Biomed..

[46]  Edwin Valarezo,et al.  Simultaneous Detection and Classification of Breast Masses in Digital Mammograms via a Deep Learning YOLO-based CAD System , 2018, Comput. Methods Programs Biomed..

[47]  V. Robert,et al.  Cophenetic correlation analysis as a strategy to select phylogenetically informative proteins: an example from the fungal kingdom , 2007, BMC Evolutionary Biology.

[48]  P. Langenberg,et al.  Breast Imaging Reporting and Data System: inter- and intraobserver variability in feature analysis and final assessment. , 2000, AJR. American journal of roentgenology.

[49]  Evgeniy Bart,et al.  Deep Synthesis of Realistic Medical Images: A Novel Tool in Clinical Research and Training , 2018, Front. Neuroinform..

[50]  Jeremy M Wolfe,et al.  Detecting the "gist" of breast cancer in mammograms three years before localized signs of cancer are visible. , 2019, The British journal of radiology.

[51]  H. Park,et al.  Changing patterns of microcalcification on screening mammography for prediction of breast cancer , 2016, Breast Cancer.

[52]  Wei Yang,et al.  Microcalcification diagnosis in digital mammography using extreme learning machine based on hidden Markov tree model of dual-tree complex wavelet transform , 2017, Expert Syst. Appl..

[53]  Trafton Drew,et al.  Scanners and drillers: characterizing expert visual search through volumetric images. , 2013, Journal of vision.

[54]  Karolina Janacsek,et al.  Predicting the future: from implicit learning to consolidation. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[55]  Alessandro Giuliani,et al.  Microcalcification morphological descriptors and parenchyma fractal dimension hierarchically interact in breast cancer: A diagnostic perspective , 2018, Comput. Biol. Medicine.

[56]  Aaron R. Seitz,et al.  A unified model for perceptual learning , 2005, Trends in Cognitive Sciences.

[57]  Karla K. Evans,et al.  The gist of the abnormal: Above-chance medical decision making in the blink of an eye , 2013, Psychonomic Bulletin & Review.

[58]  Edward K. Vogel,et al.  Neural Measures of Dynamic Changes in Attentive Tracking Load , 2012, Journal of Cognitive Neuroscience.

[59]  J. Wolfe,et al.  The Invisible Gorilla Strikes Again , 2013, Psychological science.

[60]  Valerie Andolina,et al.  Mammographic Imaging: A Practical Guide , 2001 .

[61]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[62]  Kevin W. Bowyer,et al.  Registration and difference analysis of corresponding mammogram images , 1999, Medical Image Anal..

[63]  Vyacheslav P. Tuzlukov,et al.  Signal detection theory , 2001 .

[64]  E A Krupinski,et al.  The influence of a perceptually linearized display on observer performance and visual search. , 2000, Academic radiology.

[65]  Shizhong Xu.  Principles of Statistical Genomics , 2012, Springer New York.

[66]  J. Schall,et al.  Microsaccade production during saccade cancelation in a stop-signal task , 2016, Vision Research.

[67]  Samuel E. Buttrey,et al.  Data Mining Algorithms Explained Using R , 2015 .

[68]  J. Saffran,et al.  Infant Statistical Learning , 2018, Annual review of psychology.

[69]  Karl J. Friston,et al.  Bayesian decoding of brain images , 2008, NeuroImage.

[70]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[71]  R. Shepard,et al.  Mental transformations in the identification of left and right hands. , 1975, Journal of experimental psychology. Human perception and performance.

[72]  Koenraad Verstraete,et al.  Learning from diagnostic errors: a good way to improve education in radiology. , 2011, European journal of radiology.

[73]  W. Woodward,et al.  Inflammatory breast cancer: a proposed conceptual shift in the UICC-AJCC TNM staging system. , 2017, The Lancet. Oncology.

[74]  Dev P. Chakraborty,et al.  Medical Imaging 2003: Image Perception, Observer Performance, and Technology Assessment , 2003 .

[75]  Jay Hegdé,et al.  Do different radiologists perceive medical images the same way? Some insights from Representational Similarity Analysis , 2019, HVEI.

[76]  Michael A. Cohen,et al.  The boundary conditions for Bohr’s law: when is reacting faster than acting? , 2011, Attention, perception & psychophysics.

[77]  Rita Costa,et al.  Comparison of RAPD, ISSR, and AFLP Molecular Markers to Reveal and Classify Orchardgrass (Dactylis glomerata L.) Germplasm Variations , 2016, PloS one.

[78]  Harry L. Van Trees,et al.  Detection, Estimation, and Modulation Theory, Part I , 1968 .

[79]  S. Haneuse,et al.  Educational interventions to improve screening mammography interpretation: a randomized controlled trial. , 2014, AJR. American journal of roentgenology.

[80]  R. Vogels,et al.  Inferotemporal neurons represent low-dimensional configurations of parameterized shapes , 2001, Nature Neuroscience.

[81]  Antimo Buonocore,et al.  Memory-guided microsaccades , 2019, Nature Communications.

[82]  Evgeniy Bart,et al.  Editorial: Deep Learning in Biological, Computer, and Neuromorphic Systems , 2019, Front. Comput. Neurosci..

[83]  Yongyi Yang,et al.  Quantitative comparison of clustered microcalcifications in for‐presentation and for‐processing mammograms in full‐field digital mammography , 2017, Medical physics.

[84]  T. Uematsu.  MRI findings of inflammatory breast cancer, locally advanced breast cancer, and acute mastitis: T2-weighted images can increase the specificity of inflammatory breast cancer , 2012, Breast Cancer.

[85]  Jeremy M Wolfe,et al.  A half-second glimpse often lets radiologists identify breast cancer cases even when viewing the mammogram of the opposite breast , 2016, Proceedings of the National Academy of Sciences.

[86]  C. Langlotz,et al.  Deep Learning in Neuroradiology , 2018, American Journal of Neuroradiology.

[87]  Karla K. Evans,et al.  If You Don’t Find It Often, You Often Don’t Find It: Why Some Cancers Are Missed in Breast Cancer Screening , 2013, PloS one.

[88]  Anselmo Cardoso de Paiva,et al.  Detection of mass regions in mammograms by bilateral analysis adapted to breast density using similarity indexes and convolutional neural networks , 2018, Comput. Methods Programs Biomed..

[89]  R. Shepard,et al.  Toward a universal law of generalization for psychological science. , 1987, Science.

[90]  R. Shepard,et al.  Second-order isomorphism of internal representations: Shapes of states ☆ , 1970 .

[91]  M K Markey,et al.  The reliability of measuring physical characteristics of spiculated masses on mammography. , 2006, The British journal of radiology.

[92]  Shiju Yan,et al.  A new approach to develop computer-aided diagnosis scheme of breast mass classification using deep learning technology. , 2017, Journal of X-ray science and technology.

[93]  N. Petrick,et al.  Improvement of radiologists' characterization of mammographic masses by using computer-aided diagnosis: an ROC study. , 1999, Radiology.

[94]  M. Uder,et al.  White Paper: Radiological Curriculum for Undergraduate Medical Education in Germany , 2016, RöFo - Fortschritte auf dem Gebiet der Röntgenstrahlen und der bildgebenden Verfahren.

[95]  P. Perruchet,et al.  Implicit learning and statistical learning: one phenomenon, two approaches , 2006, Trends in Cognitive Sciences.

[96]  Ashley M. Sherman,et al.  Visual search for arbitrary objects in real scenes , 2011, Attention, perception & psychophysics.

[97]  Jenny R. Saffran,et al.  Constraints on Statistical Learning Across Species , 2018, Trends in Cognitive Sciences.

[98]  Jeremy M Wolfe,et al.  Does visual expertise improve visual recognition memory? , 2011, Attention, perception & psychophysics.

[99]  Discovery Science , 2002, Lecture Notes in Computer Science.

[100]  Osamu Abe,et al.  Deep learning with convolutional neural network in radiology , 2018, Japanese Journal of Radiology.

[101]  Dev P. Chakraborty,et al.  Medical Imaging 2002: Image Perception, Observer Performance, and Technology Assessment , 2002 .

[102]  Jay Hegdé,et al.  Learning to Break Camouflage by Learning the Background , 2012, Psychological science.

[103]  J. Hegdé,et al.  Making Expert Decisions Easier to Fathom: On the Explainability of Visual Object Recognition Expertise , 2018, Front. Neurosci..

[104]  D. De Ridder,et al.  Thalamocortical dysrhythmia detected by machine learning , 2018, Nature Communications.

[105]  R. Shepard,et al.  The internal representation of numbers , 1975, Cognitive Psychology.

[106]  N. Petrick,et al.  Improvement in radiologists' characterization of malignant and benign breast masses on serial mammograms with computer-aided diagnosis: an ROC study. , 2004, Radiology.

[107]  Bahaa Ghammraoui,et al.  Investigating the feasibility of classifying breast microcalcifications using photon‐counting spectral mammography: A simulation study , 2017, Medical physics.

[108]  Adam T. Biggs,et al.  Differences in multiple-target visual search performance between non-professional and professional searchers due to decision-making criteria. , 2015, British journal of psychology.

[109]  Ashutosh Kumar Singh,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2010 .

[110]  Marimuthu Muthuvel,et al.  Microcalcification cluster detection using multiscale products based Hessian matrix via the Tsallis thresholding scheme , 2017, Pattern Recognit. Lett..

[111]  Alexander Borst,et al.  How does Nature Program Neuron Types? , 2008, Front. Neurosci..

[112]  Etta D Pisano,et al.  Diagnostic accuracy of Fischer Senoscan Digital Mammography versus screen-film mammography in a diagnostic mammography population. , 2004, Academic radiology.

[113]  Jay Hegde,et al.  Deep Visual Patterns Are Informative to Practicing Radiologists in Mammograms in Diagnostic Tasks , 2017 .

[114]  D. Halpenny,et al.  Radiological error--an early assessment of departmental radiology discrepancy meetings. , 2012, Irish medical journal.

[115]  H L Kundel,et al.  Nature of expertise in searching mammograms for breast masses , 1996, Medical Imaging.

[116]  S. Merajver,et al.  International expert panel on inflammatory breast cancer: consensus statement for standardized diagnosis and treatment. , 2011, Annals of oncology : official journal of the European Society for Medical Oncology.