A Methodological Overview on Anomaly Detection

In this Chapter we give an overview of statistical methods for anomaly detection (AD), thereby targeting an audience of practitioners with general knowledge of statistics. We focus on the applicability of the methods by stating and comparing the conditions in which they can be applied and by discussing the parameters that need to be set.

[1]  Angelo Coluccia,et al.  Distribution-based anomaly detection in 3G mobile networks: from theory to practice , 2010, Int. J. Netw. Manag..

[2]  Ronald R. Coifman,et al.  Entropy-based algorithms for best basis selection , 1992, IEEE Trans. Inf. Theory.

[3]  George Varghese,et al.  New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice , 2003, TOCS.

[4]  Nong Ye,et al.  A Markov Chain Model of Temporal Behavior for Anomaly Detection , 2000 .

[5]  M. Shyu,et al.  A Novel Anomaly Detection Scheme Based on Principal Component Classifier , 2003 .

[6]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[7]  Marina Thottan,et al.  Anomaly detection in IP networks , 2003, IEEE Trans. Signal Process..

[8]  Balachander Krishnamurthy,et al.  Sketch-based change detection: methods, evaluation, and applications , 2003, IMC '03.

[9]  Fuliang Yin,et al.  Advances in Neural Networks – ISNN 2004 , 2004, Lecture Notes in Computer Science.

[10]  Dorothy E. Denning,et al.  An Intrusion-Detection Model , 1987, IEEE Transactions on Software Engineering.

[11]  Arjun K. Gupta,et al.  Testing and Locating Variance Changepoints with Application to Stock Prices , 1997 .

[12]  Guangmin Hu,et al.  Anomaly Detection of Network Traffic Based on Wavelet Packet , 2006, 2006 Asia-Pacific Conference on Communications.

[13]  Christian Callegari,et al.  Combining sketches and wavelet analysis for multi time-scale network anomaly detection , 2011, Comput. Secur..

[14]  Mohammed J. Zaki,et al.  ADMIT: anomaly-based data mining for intrusions , 2002, KDD.

[15]  Georg Carle,et al.  Application of Forecasting Techniques and Control Charts for Traffic Anomaly Detection , 2008 .

[16]  Anil Somayaji,et al.  Analysis of the 1999 DARPA/Lincoln Laboratory IDS evaluation data with NetADHICT , 2009, 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications.

[17]  Rajeev Motwani,et al.  Approximate Frequency Counts over Data Streams , 2012, VLDB.

[18]  Moses Charikar,et al.  Finding frequent items in data streams , 2004, Theor. Comput. Sci..

[19]  D. Hsu.  Tests for Variance Shift at an Unknown Time Point , 1977 .

[20]  Michel Mandjes,et al.  Large Deviations for Gaussian Queues , 2007 .

[21]  S Matteoli,et al.  A tutorial overview of anomaly detection in hyperspectral images , 2010, IEEE Aerospace and Electronic Systems Magazine.

[22]  Alexander Ferworn,et al.  Computational public safety in emergency management communications , 2010, IWCMC.

[23]  Rajeev Rastogi,et al.  Efficient algorithms for mining outliers from large data sets , 2000, SIGMOD 2000.

[24]  Matthew Roughan,et al.  Real-time estimation of the parameters of long-range dependence , 2000, TNET.

[25]  Kensuke Fukuda,et al.  Combining sketch and wavelet models for anomaly detection , 2010, Proceedings of the 2010 IEEE 6th International Conference on Intelligent Computer Communication and Processing.

[26]  Christian Callegari,et al.  Detecting Heavy Change in the Heavy Hitter distribution of network traffic , 2011, 2011 7th International Wireless Communications and Mobile Computing Conference.

[27]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[28]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[29]  M. Pollak.  Optimal Detection of a Change in Distribution , 1985 .

[30]  Michel Mandjes,et al.  M/G/∞ transience, and its applications to overload detection , 2011, Perform. Evaluation.

[31]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[32]  Xiangliang Zhang,et al.  A Novel Intrusion Detection Method Based on Principle Component Analysis in Computer Security , 2004, ISNN.

[33]  Patrick D. McDaniel,et al.  On Attack Causality in Internet-Connected Cellular Networks , 2007, USENIX Security Symposium.

[34]  Jelena Kovacevic,et al.  Wavelets and Subband Coding , 2013, Prentice Hall Signal Processing Series.

[35]  S. Muthukrishnan,et al.  Data streams: algorithms and applications , 2005, SODA '03.

[36]  E. S. Page.  CONTINUOUS INSPECTION SCHEMES , 1954 .

[37]  G. Lorden.  PROCEDURES FOR REACTING TO A CHANGE IN DISTRIBUTION , 1971 .

[38]  Mikkel Thorup,et al.  Tabulation based 4-universal hashing with applications to second moment estimation , 2004, SODA '04.

[39]  Antonio Pescapè,et al.  NIS04-1: Wavelet-based Detection of DoS Attacks , 2006, IEEE Globecom 2006.

[40]  Anja Feldmann,et al.  A non-instrusive, wavelet-based approach to detecting network performance problems , 2001, IMW '01.

[41]  Liqun Wang,et al.  Boundary crossing probability for Brownian motion and general boundaries , 1997, Journal of Applied Probability.

[42]  Marina Vannucci,et al.  Detecting Traffic Anomalies Using Discrete Wavelet Transform , 2004, ICOIN.

[43]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[44]  Jonathon Shlens,et al.  A Tutorial on Principal Component Analysis , 2014, ArXiv.

[45]  Walter Willinger,et al.  On the self-similar nature of Ethernet traffic , 1993, SIGCOMM '93.

[46]  Kensuke Fukuda,et al.  Extracting hidden anomalies using sketch and non Gaussian multiresolution statistical detection procedures , 2007, LSAD '07.

[47]  Marco Mellia,et al.  Anomaly detection in VoIP traffic with trends , 2012, 2012 24th International Teletraffic Congress (ITC 24).

[48]  Graham Cormode,et al.  An improved data stream summary: the count-min sketch and its applications , 2004, J. Algorithms.

[49]  Yong Guan,et al.  Sketch-Based Streaming PCA Algorithm for Network-Wide Traffic Anomaly Detection , 2010, 2010 IEEE 30th International Conference on Distributed Computing Systems.

[50]  Aiko Pras,et al.  Autonomic Parameter Tuning of Anomaly-Based IDSs: an SSH Case Study , 2012, IEEE Transactions on Network and Service Management.

[51]  I. Daubechies.  Orthonormal bases of compactly supported wavelets , 1988 .

[52]  Darryl Veitch,et al.  On the Role of Flows and Sessions in Internet Traffic Modeling: An Explorative Toy-Model , 2009, GLOBECOM 2009 - 2009 IEEE Global Telecommunications Conference.

[53]  Graham Cormode,et al.  What's hot and what's not: tracking most frequent items dynamically , 2003, TODS.

[54]  Thomas F. La Porta,et al.  Security for Telecommunications Networks , 2008, Advances in Information Security.

[55]  Richard R. Brooks,et al.  Wavelet based Denial-of-Service detection , 2006, Comput. Secur..

[56]  Felix Naumann,et al.  Data fusion , 2009, CSUR.

[57]  Graham Cormode,et al.  What's new: finding significant differences in network data streams , 2004, IEEE/ACM Transactions on Networking.

[58]  Leonid Portnoy,et al.  Intrusion detection with unlabeled data using clustering , 2000 .

[59]  Yan Chen,et al.  Reversible sketches for efficient and accurate change detection over network data streams , 2004, IMC '04.

[60]  Jennifer Rexford,et al.  Sensitivity of PCA for traffic anomaly detection , 2007, SIGMETRICS '07.

[61]  Anna C. Gilbert,et al.  Multiscale Analysis and Data Networks , 2001 .

[62]  Alberto Dainotti,et al.  Wavelet-based Detection of DoS Attacks. , 2006 .

[63]  B. Brodsky,et al.  Nonparametric Methods in Change Point Problems , 1993 .

[64]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[65]  James A. Bucklew,et al.  Large Deviation Techniques in Decision, Simulation, and Estimation , 1990 .

[66]  Symeon Papavassiliou,et al.  Improving network anomaly detection effectiveness via an integrated multi-metric-multi-link (M3L) PCA-based approach , 2009, Secur. Commun. Networks.

[67]  Christopher Leckie,et al.  Adaptive Clustering for Network Intrusion Detection , 2004, PAKDD.

[68]  Konstantina Papagiannaki,et al.  Structural analysis of network traffic flows , 2004, SIGMETRICS '04/Performance '04.

[69]  Divesh Srivastava,et al.  Finding Hierarchical Heavy Hitters in Data Streams , 2003, VLDB.

[70]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[71]  S. Resnick.  Adventures in stochastic processes , 1992 .

[72]  Marina Thottan,et al.  Algorithms for Next Generation Networks , 2010, Computer Communications and Networks.

[73]  Ali A. Ghorbani,et al.  Network Anomaly Detection Based on Wavelet Analysis , 2009, EURASIP J. Adv. Signal Process..

[74]  Mark Crovella,et al.  Characterization of network-wide anomalies in traffic flows , 2004, IMC '04.

[75]  Kensuke Fukuda,et al.  Seven Years and One Day: Sketching the Evolution of Internet Traffic , 2009, IEEE INFOCOM 2009.

[76]  G. C. Tiao,et al.  Use of Cumulative Sums of Squares for Retrospective Detection of Changes of Variance , 1994 .

[77]  Songwu Lu,et al.  Securing a Wireless World , 2006, Proceedings of the IEEE.

[78]  Marina Thottan,et al.  Anomaly Detection Approaches for Communication Networks , 2010, Algorithms for Next Generation Networks.

[79]  Eleazar Eskin,et al.  A GEOMETRIC FRAMEWORK FOR UNSUPERVISED ANOMALY DETECTION: DETECTING INTRUSIONS IN UNLABELED DATA , 2002 .

[80]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[81]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD 2000.

[82]  Christian Callegari,et al.  Detecting anomalies in backbone network traffic: a performance comparison among several change detection methods , 2012, Int. J. Sens. Networks.

[83]  A. Shiryaev.  On Optimum Methods in Quickest Detection Problems , 1963 .

[84]  Christian Callegari,et al.  A novel multi time-scales PCA-based anomaly detection system , 2010, Proceedings of the 2010 International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS '10).

[85]  Anja Feldmann,et al.  On dominant characteristics of residential broadband internet traffic , 2009, IMC '09.

[86]  Soheila Dehghanzadeh,et al.  Optimizing Fuzzy K-means for network anomaly detection using PSO , 2008, 2008 IEEE/ACS International Conference on Computer Systems and Applications.

[87]  Mark Burgess,et al.  Measuring system normality , 2002, TOCS.

[88]  Wei Zhao,et al.  Adaptive CUSUM for Anomaly Detection and Its Application to Detect Shared Congestion , 2007 .

[89]  Mark Crovella,et al.  Mining anomalies using traffic feature distributions , 2005, SIGCOMM '05.

[90]  Salvatore J. Stolfo,et al.  A Geometric Framework for Unsupervised Anomaly Detection , 2002, Applications of Data Mining in Computer Security.

[91]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[92]  Christian Callegari,et al.  On the use of sketches and wavelet analysis for network anomaly detection , 2010, IWCMC.

[93]  Fabio Ricciato,et al.  A review of DoS attack models for 3G cellular networks from a system-design perspective , 2010, Comput. Commun..

[94]  A. N. Shiryaev.  On Markov Sufficient Statistics in Non-Additive Bayes Problems of Sequential Analysis , 1964 .

[95]  Christian Callegari,et al.  When randomness improves the anomaly detection performance , 2010, 2010 3rd International Symposium on Applied Sciences in Biomedical and Communication Technologies (ISABEL 2010).

[96]  Roberto Battiti,et al.  Identifying intrusions in computer networks with principal component analysis , 2006, First International Conference on Availability, Reliability and Security (ARES'06).

[97]  Christophe Diot,et al.  Diagnosing network-wide traffic anomalies , 2004, SIGCOMM.

[98]  Richard M. Karp,et al.  A simple algorithm for finding frequent elements in streams and bags , 2003, TODS.

[99]  Christian Callegari,et al.  Application of Wavelet Packet Transform to Network Anomaly Detection , 2008, NEW2AN.