Should I Raise The Red Flag? A comprehensive survey of anomaly scoring methods toward mitigating false alarms

A general Intrusion Detection System (IDS) fundamentally acts based on an Anomaly Detection System (ADS) or a combination of anomaly detection and signature-based methods, gathering and analyzing observations and reporting possible suspicious cases to a system administrator or the other users for further investigation. One of the notorious challenges which even the state-of-the-art ADS and IDS have not overcome is the possibility of a very high false alarms rate. Especially in very large and complex system settings, the amount of low-level alarms easily overwhelms administrators and increases their tendency to ignore alerts.We can group the existing false alarm mitigation strategies into two main families: The first group covers the methods directly customized and applied toward higher quality anomaly scoring in ADS. The second group includes approaches utilized in the related contexts as a filtering method toward decreasing the possibility of false alarm rates.Given the lack of a comprehensive study regarding possible ways to mitigate the false alarm rates, in this paper, we review the existing techniques for false alarm mitigation in ADS and present the pros and cons of each technique. We also study a few promising techniques applied in the signature-based IDS and other related contexts like commercial Security Information and Event Management (SIEM) tools, which are applicable and promising in the ADS context.Finally, we conclude with some directions for future research.

[1]  Ville Ollikainen,et al.  A new similarity measure using Bhattacharyya coefficient for collaborative filtering in sparse data , 2015, Knowl. Based Syst..

[2]  Sandro Etalle,et al.  ATLANTIDES: An Architecture for Alert Verification in Network Intrusion Detection Systems , 2007, LISA.

[3]  Reda Alhajj,et al.  A comprehensive survey of numeric and symbolic outlier mining techniques , 2006, Intell. Data Anal..

[4]  Valentino Constantinou,et al.  Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding , 2018, KDD.

[5]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[6]  Shubhomoy Das,et al.  Active Anomaly Detection via Ensembles , 2018, ArXiv.

[7]  Yue Zhao,et al.  PyOD: A Python Toolbox for Scalable Outlier Detection , 2019, J. Mach. Learn. Res..

[8]  Zhi-Hua Zhou,et al.  Isolation Forest , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[9]  Tadeusz Pietraszek,et al.  Data mining and machine learning - Towards reducing false positives in intrusion detection , 2005, Inf. Secur. Tech. Rep..

[10]  Jinoh Kim,et al.  A survey of deep learning-based network anomaly detection , 2017, Cluster Computing.

[11]  Edwin Lughofer,et al.  Hybrid and Ensemble Methods in Machine Learning , 2013, J. Univers. Comput. Sci..

[12]  Peng Ning,et al.  Analyzing Intensive Intrusion Alerts via Correlation , 2002, RAID.

[13]  Mehdi Khashei,et al.  A novel hybridization of artificial neural networks and ARIMA models for time series forecasting , 2011, Appl. Soft Comput..

[14]  Nicolas Goix,et al.  How to Evaluate the Quality of Unsupervised Anomaly Detection Algorithms? , 2016, ArXiv.

[15]  Sridhar Adepu,et al.  Anomaly Detection in Cyber Physical Systems Using Recurrent Neural Networks , 2017, 2017 IEEE 18th International Symposium on High Assurance Systems Engineering (HASE).

[16]  Dong Xiang,et al.  Information-theoretic measures for anomaly detection , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[17]  Douglas M. Hawkins.  Identification of Outliers , 1980, Monographs on Applied Probability and Statistics.

[18]  Mia Hubert,et al.  Robust statistics for outlier detection , 2011, WIREs Data Mining Knowl. Discov..

[19]  Alexandre Termier,et al.  Anomaly Detection in Streams with Extreme Value Theory , 2017, KDD.

[20]  Hari Om,et al.  A hybrid system for reducing the false alarm rate of anomaly intrusion detection system , 2012, 2012 1st International Conference on Recent Advances in Information Technology (RAIT).

[21]  Neminath Hubballi,et al.  False alarm minimization techniques in signature-based intrusion detection systems: A survey , 2014, Comput. Commun..

[22]  Clayton D. Scott,et al.  Robust kernel density estimation , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[23]  R. McCrae.  Creativity, divergent thinking, and openness to experience. , 1987 .

[24]  Indre Zliobaite,et al.  Learning under Concept Drift: an Overview , 2010, ArXiv.

[25]  Christopher Leckie,et al.  High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning , 2016, Pattern Recognit..

[26]  Çagdas Hakan Aladag,et al.  Fuzzy time series forecasting with a novel hybrid approach combining fuzzy c-means and neural networks , 2013, Expert Syst. Appl..

[27]  Pedro M. Domingos,et al.  Sum-product networks: A new deep architecture , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[28]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[29]  Jerzy W. Rozenblit,et al.  Alert Fusion for a Computer Host Based Intrusion Detection System , 2007, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07).

[30]  Thomas G. Dietterich,et al.  Systematic construction of anomaly detection benchmarks from real data , 2013, ODD '13.

[31]  William K. Robertson,et al.  Beehive: large-scale log analysis for detecting suspicious activity in enterprise networks , 2013, ACSAC.

[32]  Subutai Ahmad,et al.  Unsupervised real-time anomaly detection for streaming data , 2017, Neurocomputing.

[33]  Sungzoon Cho,et al.  Variational Autoencoder based Anomaly Detection using Reconstruction Probability , 2015 .

[34]  Robert A. Bridges,et al.  Setting the threshold for high throughput detectors: A mathematical approach for ensembles of dynamic, heterogeneous, probabilistic anomaly detectors , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[35]  Rafal Pokrywka.  Reducing False Alarm Rate in Anomaly Detection with Layered Filtering , 2008, ICCS.

[36]  Thomas G. Dietterich,et al.  A Meta-Analysis of the Anomaly Detection Problem , 2015 .

[37]  Thomas J. Veasey,et al.  Anomaly Detection in Application Performance Monitoring Data , 2014 .

[38]  Jung-Min Park,et al.  An overview of anomaly detection techniques: Existing solutions and latest technological trends , 2007, Comput. Networks.

[39]  Uwe Glässer,et al.  Hidden Markov based anomaly detection for water supply systems , 2016, 2016 IEEE International Conference on Big Data (Big Data).

[40]  Milos Manic,et al.  Toward Explainable Deep Neural Network Based Anomaly Detection , 2018, 2018 11th International Conference on Human System Interaction (HSI).

[41]  Elena Deza,et al.  Encyclopedia of Distances , 2014 .

[42]  Vipin Kumar,et al.  Comparative Evaluation of Anomaly Detection Techniques for Sequence Data , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[43]  Christoph Meinel,et al.  A New Alert Correlation Algorithm Based on Attack Graph , 2011, CISIS.

[44]  Peter J. Rousseeuw,et al.  Robust Regression and Outlier Detection , 2005, Wiley Series in Probability and Statistics.

[45]  Zhilin Li,et al.  A Multiscale Approach for Spatio‐Temporal Outlier Detection , 2006, Trans. GIS.

[46]  Marina L. Gavrilova,et al.  Artificial Face Recognition Using Wavelet Adaptive LBP with Directional Statistical Features , 2012, 2012 International Conference on Cyberworlds.

[47]  Sanjay Chawla,et al.  Mining for Outliers in Sequential Databases , 2006, SDM.

[48]  Stephan Cl'emenccon,et al.  Mass Volume Curves and Anomaly Ranking , 2017, 1705.01305.

[49]  Takehisa Yairi,et al.  Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction , 2014, MLSDA'14.

[50]  Dominique T. Shipmon,et al.  Time Series Anomaly Detection; Detection of anomalous drops with limited features and sparse examples in noisy highly periodic data , 2017, ArXiv.

[51]  John Cristian Borges Gamboa,et al.  Deep Learning for Time-Series Analysis , 2017, ArXiv.

[52]  Uwe Glässer,et al.  Dynamic Attack Scoring Using Distributed Local Detectors , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[53]  Brett J. Borghetti,et al.  A Survey of Distance and Similarity Measures Used Within Network Intrusion Anomaly Detection , 2015, IEEE Communications Surveys & Tutorials.

[54]  R. Serfling,et al.  General notions of statistical depth function , 2000 .

[55]  E. S. Page.  CONTINUOUS INSPECTION SCHEMES , 1954 .

[56]  Barry E. Mullins,et al.  Alert Verification Evasion Through Server Response Forging , 2007, RAID.

[57]  M.M. Deris,et al.  A Comparative Study for Outlier Detection Techniques in Data Mining , 2006, 2006 IEEE Conference on Cybernetics and Intelligent Systems.

[58]  Asaf Shabtai,et al.  Detecting Cyber Attacks in Industrial Control Systems Using Convolutional Neural Networks , 2018, CPS-SPC@CCS.

[59]  Priya Narasimhan,et al.  Tiresias: Black-Box Failure Prediction in Distributed Systems , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[60]  Aderemi Oluyinka Adewumi,et al.  A survey of machine-learning and nature-inspired based credit card fraud detection techniques , 2016, International Journal of System Assurance Engineering and Management.

[61]  Klaus Julisch,et al.  Using root cause analysis to handle intrusion detection alarms , 2003 .

[62]  Subutai Ahmad,et al.  Evaluating Real-Time Anomaly Detection Algorithms -- The Numenta Anomaly Benchmark , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[63]  Irad Ben-Gal.  Outlier Detection , 2005, The Data Mining and Knowledge Discovery Handbook.

[64]  Stan Matwin,et al.  Fast Unsupervised Online Drift Detection , 2016, Knowledge Discovery and Data Mining.

[65]  Lovekesh Vig,et al.  Long Short Term Memory Networks for Anomaly Detection in Time Series , 2015, ESANN.

[66]  Uwe Glässer,et al.  Deep Learning Based Forecasting of Critical Infrastructure Data , 2017, CIKM.

[67]  Wenke Lee,et al.  Discovering Novel Attack Strategies from INFOSEC Alerts , 2004, ESORICS.

[68]  Jérémie Jakubowicz,et al.  Scoring anomalies: a M-estimation formulation , 2013, AISTATS.

[69]  Jason Lee,et al.  The NIDS Cluster: Scalable, Stateful Network Intrusion Detection on Commodity Hardware , 2007, RAID.

[70]  Žliobait . e,et al.  Learning under Concept Drift: an Overview , 2010 .

[71]  Giovanni Vigna,et al.  A stateful intrusion detection system for World-Wide Web servers , 2003, 19th Annual Computer Security Applications Conference, 2003. Proceedings..

[72]  Robert A. Bridges,et al.  A New, Principled Approach to Anomaly Detection , 2012, 2012 11th International Conference on Machine Learning and Applications.

[73]  Mohiuddin Ahmed,et al.  Thwarting DoS Attacks: A Framework for Detection based on Collective Anomalies and Clustering , 2017, Computer.

[74]  Leonid Portnoy,et al.  Intrusion detection with unlabeled data using clustering , 2000 .

[75]  David J. Hill,et al.  Anomaly detection in streaming environmental sensor data: A data-driven modeling approach , 2010, Environ. Model. Softw..

[76]  Vern Paxson,et al.  Outside the Closed World: On Using Machine Learning for Network Intrusion Detection , 2010, 2010 IEEE Symposium on Security and Privacy.

[77]  Mohsen Guizani,et al.  Deep Learning for IoT Big Data and Streaming Analytics: A Survey , 2017, IEEE Communications Surveys & Tutorials.

[78]  Peter J. Rousseeuw,et al.  Robust regression and outlier detection , 1987 .

[79]  Alfonso Valdes,et al.  An Approach to Sensor Correlation , 2000 .

[80]  Charu C. Aggarwal,et al.  Outlier Detection for Temporal Data: A Survey , 2014, IEEE Transactions on Knowledge and Data Engineering.

[81]  Marc'Aurelio Ranzato,et al.  Efficient Learning of Sparse Representations with an Energy-Based Model , 2006, NIPS.

[82]  Hossein Mobahi,et al.  Deep Learning via Semi-supervised Embedding , 2012, Neural Networks: Tricks of the Trade.

[83]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[84]  Xi Chen,et al.  Direct Robust Matrix Factorizatoin for Anomaly Detection , 2011, 2011 IEEE 11th International Conference on Data Mining.

[85]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[86]  Thomas M. Arnold.  Quantifying the Behavior of Stock Correlations under Market Stress , 2013 .

[87]  Lionel Tarassenko,et al.  Static and dynamic novelty detection methods for jet engine health monitoring , 2007, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[88]  Lance Sherry,et al.  Anomaly detection in aircraft data using Recurrent Neural Networks (RNN) , 2016, 2016 Integrated Communications Navigation and Surveillance (ICNS).

[89]  Salvatore J. Stolfo,et al.  Anomalous Payload-Based Network Intrusion Detection , 2004, RAID.

[90]  Minrui Fei,et al.  An Anomaly Detection Approach Based on Isolation Forest Algorithm for Streaming Data Using Sliding Window , 2013, ICONS.

[91]  Philip K. Chan,et al.  Tracking User Mobility to Detect Suspicious Behavior , 2009, SDM.

[92]  Georg Langs,et al.  Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery , 2017, IPMI.

[93]  Hans-Peter Kriegel,et al.  A survey on unsupervised outlier detection in high‐dimensional numerical data , 2012, Stat. Anal. Data Min..

[94]  R. Fisher,et al.  Limiting forms of the frequency distribution of the largest or smallest member of a sample , 1928, Mathematical Proceedings of the Cambridge Philosophical Society.

[95]  Ke Wang,et al.  Contextual verification for false alarm reduction in maritime anomaly detection , 2015, 2015 IEEE International Conference on Big Data (Big Data).

[96]  Seiichi Uchida,et al.  A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data , 2016, PloS one.

[97]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[98]  Mansoor Alam,et al.  A Deep Learning Approach for Network Intrusion Detection System , 2016, EAI Endorsed Trans. Security Safety.

[99]  M. Schervish.  P Values: What They are and What They are Not , 1996 .

[100]  Ali A. Ghorbani,et al.  An incremental frequent structure mining framework for real-time alert correlation , 2009, Comput. Secur..

[101]  Hans-Peter Kriegel,et al.  Interpreting and Unifying Outlier Scores , 2011, SDM.

[102]  K. P. Soman,et al.  Applying convolutional neural network for network intrusion detection , 2017, 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI).