On Evaluation of Outlier Rankings and Outlier Scores

Outlier detection research is currently focusing on the development of new methods and on improving the computation time for these methods. Evaluation however is rather heuristic, often considering just precision in the top k results or using the area under the ROC curve. These evaluation procedures do not allow for assessment of similarity between methods. Judging the similarity of or correlation between two rankings of outlier scores is an important question in itself but it is also an essential step towards meaningfully building outlier detection ensembles, where this aspect has been completely ignored so far. In this study, our generalized view of evaluation methods allows both to evaluate the performance of existing methods as well as to compare different methods w.r.t. their detection performance. Our new evaluation framework takes into consideration the class imbalance problem and offers new insights on similarity and redundancy of existing outlier detection methods. As a result, the design of effective ensemble methods for outlier detection is considerably enhanced.

[1]  Ian Soboroff,et al.  Ranking retrieval systems without relevance judgments , 2001, SIGIR '01.

[2]  Hans-Peter Kriegel,et al.  Angle-based outlier detection in high-dimensional data , 2008, KDD.

[3]  Vivekanand Gopalkrishnan,et al.  Efficient Pruning Schemes for Distance-Based Outlier Detection , 2009, ECML/PKDD.

[4]  Roger Newson,et al.  Parameters behind “Nonparametric” Statistics: Kendall's tau, Somers’ D and Median Differences , 2002 .

[5]  Ezz H. Abdelfattah,et al.  A New Weighted Rank Correlation , 2008 .

[6]  Sanjay Chawla,et al.  Finding Local Anomalies in Very High Dimensional Space , 2010, 2010 IEEE International Conference on Data Mining.

[7]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[8]  Jing Gao,et al.  Converting Output Scores from Outlier Detection Algorithms into Probability Estimates , 2006, Sixth International Conference on Data Mining (ICDM'06).

[9]  Thomas Seidl,et al.  ClasSi: Measuring Ranking Quality in the Presence of Object Classes with Similarity Information , 2011, PAKDD Workshops.

[10]  Shengli Wu,et al.  Methods for ranking information retrieval systems without relevance judgments , 2003, SAC '03.

[11]  Stephen D. Bay,et al.  Mining distance-based outliers in near linear time with randomization and a simple pruning rule , 2003, KDD '03.

[12]  Ben Carterette,et al.  On rank correlation and the distance between rankings , 2009, SIGIR.

[13]  Ke Zhang,et al.  A New Local Distance-Based Outlier Detection Approach for Scattered Real-World Data , 2009, PAKDD.

[14]  Vipin Kumar,et al.  Feature bagging for outlier detection , 2005, KDD '05.

[15]  Sergei Vassilvitskii,et al.  Generalized distances between rankings , 2010, WWW '10.

[16]  Moni Naor,et al.  Optimal aggregation algorithms for middleware , 2001, PODS.

[17]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[18]  Stephen E. Robertson,et al.  A new rank correlation coefficient for information retrieval , 2008, SIGIR '08.

[19]  Elke Achtert,et al.  Spatial Outlier Detection: Data, Algorithms, Visualizations , 2011, SSTD.

[20]  Stan Matwin,et al.  Smooth Receiver Operating Characteristics (smROC) Curves , 2011, ECML/PKDD.

[21]  Massimo Melucci,et al.  On rank correlation in information retrieval evaluation , 2007, SIGF.

[22]  C. Spearman.  The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[23]  Hans-Peter Kriegel,et al.  LoOP: local outlier probabilities , 2009, CIKM.

[24]  Surya Nepal,et al.  Query processing issues in image (multimedia) databases , 1999, Proceedings 15th International Conference on Data Engineering (Cat. No.99CB36337).

[25]  Arnold W. M. Smeulders,et al.  The Amsterdam Library of Object Images , 2004, International Journal of Computer Vision.

[26]  Sridhar Ramaswamy,et al.  Efficient algorithms for mining outliers from large data sets , 2000, SIGMOD '00.

[27]  Emmanuel Müller,et al.  Statistical selection of relevant subspace projections for outlier ranking , 2011, 2011 IEEE 27th International Conference on Data Engineering.

[28]  Grace S. Shieh.  A weighted Kendall's tau statistic , 1998 .

[29]  Anthony K. H. Tung,et al.  Mining top-n local outliers in large databases , 2001, KDD '01.

[30]  Ronald Fagin,et al.  Comparing top k lists , 2003, SODA '03.

[31]  Ali S. Hadi,et al.  Detection of outliers , 2009 .

[32]  Raymond T. Ng,et al.  Distance-based outliers: algorithms and applications , 2000, The VLDB Journal.

[33]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[34]  Srinivasan Parthasarathy,et al.  Distance-based outlier detection , 2010, Proc. VLDB Endow..

[35]  D. Blest.  Theory & Methods: Rank Correlation — an Alternative Measure , 2000 .

[36]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[37]  Giorgio Valentini,et al.  Ensembles of Learning Machines , 2002, WIRN.

[38]  Joydeep Ghosh,et al.  Cluster ensembles , 2011, Data Clustering: Algorithms and Applications.

[39]  Hans-Peter Kriegel,et al.  Interpreting and Unifying Outlier Scores , 2011, SDM.

[40]  Vivekanand Gopalkrishnan,et al.  Mining Outliers with Ensemble of Heterogeneous Detectors on Random Subspaces , 2010, DASFAA.

[41]  Ronald Fagin,et al.  Combining Fuzzy Information from Multiple Systems , 1999, J. Comput. Syst. Sci..

[42]  Bernhard Liebl,et al.  Very high compliance in an expanded MS-MS-based newborn screening program despite written parental consent. , 2002, Preventive medicine.

[43]  Klemens Böhm,et al.  HiCS: High Contrast Subspaces for Density-Based Outlier Ranking , 2012, 2012 IEEE 28th International Conference on Data Engineering.

[44]  J. Rodgers,et al.  Thirteen ways to look at the correlation coefficient , 1988 .

[45]  Clara Pizzuti,et al.  Fast Outlier Detection in High Dimensional Spaces , 2002, PKDD.