Bootstrap Variability Studies in ROC Analysis on Large Datasets

The nonparametric two-sample bootstrap is employed to compute uncertainties of measures in receiver operating characteristic (ROC) analysis on large datasets in areas such as biometrics, and so on. In this framework, the bootstrap variability was empirically studied without a normality assumption, exhaustively in five scenarios involving both high- and low-accuracy matching algorithms. With a tolerance 0.02 of the coefficient of variation, it was found that 2000 bootstrap replications were appropriate for ROC analysis on large datasets in order to reduce the bootstrap variance and ensure the accuracy of the computation.

[1]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[2]  Rob J Hyndman,et al.  Sample Quantiles in Statistical Packages , 1996 .

[3]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[4]  P. Hall.  On the Number of Bootstrap Simulations Required to Construct a Confidence Interval , 1986 .

[5]  Raghu N. Kacker,et al.  Further studies of bootstrap variability for ROC analysis on large datasets , 2010 .

[6]  R. F. Brown,et al.  PERFORMANCE EVALUATION , 2019, ISO 22301:2019 and business continuity management – Understand how to plan, implement and enhance a business continuity management system (BCMS).

[7]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[8]  Raghu N. Kacker,et al.  Data dependency on measurement uncertainties in speaker recognition evaluation , 2012, Defense, Security, and Sensing.

[9]  Anil K. Jain,et al.  Performance evaluation of fingerprint verification systems , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  B. Efron.  Better Bootstrap Confidence Intervals , 1987 .

[11]  Raghu N. Kacker,et al.  Measures, Uncertainties, and Significance Test in Operational ROC Analysis , 2011, Journal of research of the National Institute of Standards and Technology.

[12]  Charles L. Wilson,et al.  An empirical study of sample size in ROC-curve analysis of fingerprint data , 2006, SPIE Defense + Commercial Sensing.

[13]  Charles L. Wilson,et al.  Nonparametric analysis of fingerprint data on large data sets , 2007, Pattern Recognit..