Evaluation of diagnostic performance using partial area under the roc curve

Evaluation of diagnostic performance is critical in many fields including but not limited to diagnostic medicine. The Receiver Operating Characteristic (ROC) curve is the most widely used methodology for describing the intrinsic performance of diagnostic tests, with the area under the curve (AUC) being the most commonly used summary index of overall performance. The partial area under the ROC curve (pAUC), when focused on the range of practical/clinical relevance, is considered a more relevant summary index than the full AUC. However, several conceptual and analytical difficulties frequently prevent the pAUC from being used. First, in many diagnostic setting the relevant range is difficult to determine objectively. Second, in theory, due to potential use of less information, analysis based on the pAUC could lead to the loss of statistical precision and therefore would require larger sample sizes. Through mathematical derivation, extensive simulation studies and practical examples, this work investigates statistical properties when using the pAUC. First, this work demonstrates that in many practical scenarios inferences based on pAUC could be more powerful than inferences based on the full AUC. Thus, the use of the pAUC may lead to not only more clinically relevant but also more conclusive results in analyses of experimental data. Second, this investigation demonstrates that the advantages of pAUC-based inferences depend on the shape of ROC curves. The conventional binormal model does not always adequately describe scenarios where the pAUC is more statistically efficient. The bi-gamma family of concave ROC curves is shown to describe practically reasonable scenarios where either pAUC or full AUC could be advantageous. Programs for sample size estimation based on bi-gamma model are then developed. Finally, this work investigates the properties of pAUC-based inferences in scenarios where diagnostic results have substantial ties (or a "mass") at the lowest diagnostic results. For certain type of the ROC curves the existence of ties could lead to an increase in statistical efficiency. Forcing a diagnostic system to resolve ties could detrimentally affect reliability and conclusiveness of statistical inferences. In conclusion, this work provides investigators with insights into and tools for generating practically relevant conclusions using pAUC. The public health importance of this work stems from the relevance of the ROC analysis at different stages of development and regulatory approval of diagnostic systems in medicine. Enhanced methodology for evaluation of diagnostic accuracy helps in the development of improved diagnostic systems and could accelerate the delivery and clinical adoption of truly beneficial diagnostic technologies and/or clinical practices.