Evaluation and amelioration of computer-aided diagnosis with artificial neural networks utilizing small-sized sample sets

Abstract Concerns about the specificity and reliability of artificial neural networks (ANNs) impede further application of ANNs in medicine. This is particularly true when developing computer-aided diagnosis (CAD) tools using ANNs for orphan diseases and emerging research areas where only a small-sized sample set is available. It is unreasonable to claim one ANN's performance as better than another simply on the basis of a single output without considering possible output variability due to factors including data noise and ANN training protocols. In this paper, a bootstrap resampling method is proposed to quantitatively analyze ANN output reliability and changing performance as the sample data and training protocols are varied. The method is tested in the area of feature classification for analysis of masses detected on mammograms. Our experiments show that ANNs performance, measured in terms of the area under the receiver operating characteristic (ROC) curve, is not a fixed value, but follows a distribution function sensitive to many factors. We demonstrate that our approach to determining the bootstrap estimates of confidence intervals (CIs) and prediction intervals (PIs) can be used to assure optimal performance in terms of ANN model configuration. We also show that the unintentional inclusion of data noise, which biases ANN results in small task-specific databases, can be accurately detected via the bootstrap estimates.

[1]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[2]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[3]  Steven K. Rogers,et al.  Neural network Bayes error estimation , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[4]  Hussein Alnuweiri,et al.  Acceleration of back propagation through initial weight pre-training with delta rule , 1993, IEEE International Conference on Neural Networks.

[5]  C. Metz.  Basic principles of ROC analysis. , 1978, Seminars in nuclear medicine.

[6]  Rangaraj M. Rangayyan,et al.  Content-based retrieval and analysis of mammographic masses , 2005, J. Electronic Imaging.

[7]  Saeid Nahavandi,et al.  Developing optimal neural network metamodels based on prediction intervals , 2009, 2009 International Joint Conference on Neural Networks.

[8]  Michael R Chernick,et al.  Bootstrap Methods: A Guide for Practitioners and Researchers , 2007 .

[9]  Myung Won Kim,et al.  The effect of initial weights on premature saturation in back-propagation learning , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[10]  Adam Blum,et al.  Neural Networks in C++: An Object-Oriented Framework for Building Connectionist Systems , 1992 .

[11]  Rangaraj M. Rangayyan,et al.  Classification of tumors and masses in mammograms using neural networks with shape and texture features , 2003, Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439).

[12]  R. Rangayyan,et al.  Boundary modelling and shape analysis methods for classification of mammographic masses , 2000, Medical and Biological Engineering and Computing.

[13]  Y. Liu,et al.  The application of Efron's bootstrap methods in validating feature classification using artificial neural networks for the analysis of mammographic masses , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[14]  Abdelhak M. Zoubir,et al.  Bootstrap techniques for signal processing , 2004 .

[15]  Alan F. Murray,et al.  Confidence estimation methods for neural networks : a practical comparison , 2001, ESANN.

[16]  Michael R. Smith,et al.  Evaluation of Several Nonparametric Bootstrap Methods to Estimate Confidence Intervals for Software Metrics , 2003, IEEE Trans. Software Eng..

[17]  M. Zweig,et al.  Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. , 1993, Clinical chemistry.

[18]  Rangaraj M. Rangayyan,et al.  Classification of breast masses in mammograms using neural networks with shape, edge sharpness, and texture features , 2006, J. Electronic Imaging.

[19]  Tong Lin,et al.  低ビットレートビデオ会議に対する関心領域利用レート制御 | 文献情報 | J-GLOBAL 科学技術総合リンクセンター , 2006 .

[20]  Wu Guo-Zua,et al.  H.264ビデオのロバスト電子透かし埋め込み/検出アルゴリズム , 2005 .

[21]  Georges Dupret,et al.  Bootstrap re-sampling for unbalanced data in supervised learning , 2001, Eur. J. Oper. Res..

[22]  Rangaraj M. Rangayyan,et al.  Evaluation of the sensitivity of a medical data-mining application to the number of elements in small databases , 2009, Biomed. Signal Process. Control..

[23]  Kevin N. Gurney,et al.  An introduction to neural networks , 2018 .

[24]  Saeid Nahavandi,et al.  Optimizing the quality of bootstrap-based prediction intervals , 2011, The 2011 International Joint Conference on Neural Networks.

[25]  J Carpenter,et al.  Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. , 2000, Statistics in medicine.

[26]  Rangaraj M. Rangayyan,et al.  An indexed atlas of digital mammograms for computer-aided diagnosis of breast cancer , 2003, Ann. des Télécommunications.

[27]  R. Nakano,et al.  Estimating expected error rates of neural network classifiers in small sample size situations: a comparison of cross-validation and bootstrap , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[28]  B. Efron.  Bootstrap Methods: Another Look at the Jackknife , 1979 .

[29]  Abdefihak M. Zoubir,et al.  Bootstrap Methods and Applications , 2007, IEEE Signal Processing Magazine.

[30]  Yoshihiko Hamamoto,et al.  Use of bootstrap samples in designing artificial neural network classifiers , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[31]  A. K. Pujari,et al.  Data Mining Techniques , 2006 .

[32]  Boualem Boashash,et al.  The bootstrap and its application in signal processing , 1998, IEEE Signal Process. Mag..

[33]  H. Guterman,et al.  Knowledge extraction from artificial neural network models , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[34]  Voicu Groza,et al.  Confidence Interval Estimation for Oscillometric Blood Pressure Measurements Using Bootstrap Approaches , 2011, IEEE Transactions on Instrumentation and Measurement.

[35]  Yulei Jiang.  Uncertainty in the output of artificial neural networks , 2003 .