Evaluation of Anomaly Detection Algorithms for the Real-World Applications

Anomaly detection in complex data structures is one of the most challenging problems in computer vision. In many real-world problems, for example in the quality control in modern manufacturing, the anomalous samples are usually rare, resulting in (highly) imbalanced datasets. However, in current research practice, these scenarios are rarely modeled, and as a consequence, evaluation of anomaly detection algorithms often do not reproduce results that are useful for practical applications. First, even in case of highly unbalanced input data, anomaly detection algorithms are expected to significantly reduce the proportion of anomalous samples, detecting “almost all” anomalous samples (with exact specifications depending on the target customer). This places high importance on only the small part of the ROC curve, possibly rendering the standard metrics such as AUC (Area Under Curve) and AP (Average Precision) useless. Second, the target of automatic anomaly detection in practical applications is significant reduction in manual work required, and standard metrics are poor predictor of this feature. Finally, the evaluation may produce erratic results for different randomly initialized training runs of the neural network, producing evaluation results that may not reproduce well in practice. In this paper, we present an evaluation methodology that avoids these pitfalls.