An overview and a benchmark of active learning for outlier detection with one-class classifiers

Active learning methods increase classification quality by means of user feedback. An important subcategory is active learning for outlier detection with one-class classifiers. While various methods in this category exist, selecting one for a given application scenario is difficult. This is because existing methods rely on different assumptions, have different objectives, and often are tailored to a specific use case. All this calls for a comprehensive comparison, the topic of this article. This article starts with a categorization of the various methods. We then propose ways to evaluate active learning results. Next, we run extensive experiments to compare existing methods, for a broad variety of scenarios. Based on our results, we formulate guidelines on how to select active learning methods for outlier detection with one-class classifiers.

[1]  Hema Raghavan,et al.  Active Learning with Feedback on Features and Instances , 2006, J. Mach. Learn. Res..

[2]  Marius Kloft,et al.  Active learning for network intrusion detection , 2009, AISec '09.

[3]  John Platt,et al.  ALADIN: Active Learning of Anomalies to Detect Intrusion , 2008 .

[4]  Bernhard Sick,et al.  Limitations of Assessing Active Learning Performance at Runtime , 2019, ArXiv.

[5]  Vincent Lemaire,et al.  How to select information that matters: a comparative study on active learning strategies for classification , 2015, I-KNOW.

[6]  Qiang Yang,et al.  One-Class Collaborative Filtering , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[7]  Abdulrahman H. Altalhi,et al.  Statistical comparisons of active learning strategies over multiple datasets , 2018, Knowl. Based Syst..

[8]  Robert P. W. Duin,et al.  Selective Sampling Methods in One-Class Classification Problems , 2003, ICANN.

[9]  Maria Eugenia Ramirez-Loaiza,et al.  Active learning: an empirical study of common baselines , 2017, Data Mining and Knowledge Discovery.

[10]  Bernhard Schölkopf,et al.  One-Class Support Measure Machines for Group Anomaly Detection , 2013, UAI.

[11]  Arthur Zimek,et al.  On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study , 2016, Data Mining and Knowledge Discovery.

[12]  Brian Mac Namee,et al.  Model-Free and Model-Based Active Learning for Regression , 2016, UKCI.

[13]  Shehroz S. Khan,et al.  One-class classification: taxonomy of study and review of techniques , 2013, The Knowledge Engineering Review.

[14]  Bernhard Sick,et al.  Challenges of Reliable, Realistic and Comparable Active Learning Evaluation , 2017, IAL@PKDD/ECML.

[15]  Zachary Chase Lipton.  The mythos of model interpretability , 2016, ACM Queue.

[16]  Lin Zhang,et al.  Two methods of selecting Gaussian kernel parameters for one-class SVM and their application to fault detection , 2014, Knowl. Based Syst..

[17]  Shuming Li,et al.  A Lightweight Anomaly Detection Method Based on SVDD for Wireless Sensor Networks , 2019, Wireless Personal Communications.

[18]  Alan Edelman,et al.  Julia: A Fresh Approach to Numerical Computing , 2014, SIAM Rev..

[19]  Thomas G. Dietterich,et al.  Incorporating Expert Feedback into Active Anomaly Discovery , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[20]  Nathalie Japkowicz,et al.  Active Learning for One-Class Classification , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[21]  Kristian Kersting,et al.  "Why Should I Trust Interactive Learners?" Explaining Interactive Queries of Classifiers to Users , 2018, ArXiv.

[22]  Richard L. Phillips,et al.  Interpretable Active Learning , 2018, FAT.

[23]  Ira Assent,et al.  Explaining Outliers by Subspace Separability , 2013, 2013 IEEE 13th International Conference on Data Mining.

[24]  Burr Settles,et al.  Active Learning , 2012, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[25]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD 2000.

[26]  Hamid R. Rabiee,et al.  ACTIVE ONE-CLASS LEARNING BY KERNEL DENSITY ESTIMATION , 2011 .

[27]  Hans-Peter Kriegel,et al.  Angle-based outlier detection in high-dimensional data , 2008, KDD.

[28]  Francisco Herrera,et al.  Instance reduction for one-class classification , 2018, Knowledge and Information Systems.

[29]  András Kocsor,et al.  Counter-Example Generation-Based One-Class Classification , 2007, ECML.

[30]  Marius Kloft,et al.  Toward Supervised Anomaly Detection , 2014, J. Artif. Intell. Res..

[31]  Hamid R. Rabiee,et al.  Active Learning from Positive and Unlabeled Data , 2011, 2011 IEEE 11th International Conference on Data Mining Workshops.

[32]  Lili Yin,et al.  Active learning based support vector data description method for robust novelty detection , 2018, Knowl. Based Syst..

[33]  Robert P. W. Duin,et al.  Uniform Object Generation for Optimizing One-class Classifiers , 2002, J. Mach. Learn. Res..

[34]  Yuhua Li,et al.  Selecting training points for one-class support vector machines , 2011, Pattern Recognit. Lett..

[35]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[36]  Robert P. W. Duin,et al.  Uncertainty sampling methods for one-class classifiers , 2003 .

[37]  Xiaoli Li,et al.  Learning from Positive and Unlabeled Examples with Different Data Distributions , 2005, ECML.

[38]  Robert P. W. Duin,et al.  Support Vector Data Description , 2004, Machine Learning.

[39]  Joseph Y. Lo,et al.  Anomaly detection for medical images based on a one-class classification , 2018, Medical Imaging.

[40]  Martin Müller,et al.  Towards User‐Centered Active Learning Algorithms , 2018, Comput. Graph. Forum.

[41]  Charu C. Aggarwal,et al.  Outlier Analysis , 2013, Springer New York.

[42]  Fredrik Olsson,et al.  A literature survey of active machine learning in the context of natural language processing , 2009 .

[43]  Andrew McCallum,et al.  Active Learning by Labeling Features , 2009, EMNLP.

[44]  Qiang Liu,et al.  Hyperparameter selection of one-class support vector machine by self-adaptive data shifting , 2018, Pattern Recognit..

[45]  Piotr Juszczak.  Learning to recognise : a study on one-class classification and active learning , 2006 .

[46]  Gavin C. Cawley,et al.  Baseline Methods for Active Learning , 2011, Active Learning and Experimental Design @ AISTATS.

[47]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[48]  Klaus-Robert Müller,et al.  Towards Explaining Anomalies: A Deep Taylor Decomposition of One-Class Models , 2018, Pattern Recognit..

[49]  Jay Lee,et al.  A modified support vector data description based novelty detection approach for machinery components , 2013, Appl. Soft Comput..