Topological Data Analysis for Enhancing Embedded Analytics for Enterprise Cyber Log Analysis and Forensics

Forensic analysis of logs is one responsibility of an enterprise cyber defense team; inherently, this is a big data task with thousands of events possibly logged in minutes of activity. Logged events range from authorized users typing incorrect passwords to malignant threats. Log analysis is necessary to understand current threats, be proactive against emerging threats, and develop new firewall rules. This paper describes embedded analytics for log analysis, which incorporates five mechanisms: numerical, similarity, graph-based, graphical analysis, and interactive feedback. Topological Data Analysis (TDA) is introduced for log analysis with TDA providing novel graph-based similarity understanding of threats which additionally enables a feedback mechanism to further analyze log files. Using real-world firewall log data from an enterprise-level organization, our end-to-end evaluation shows the effective detection and interpretation of log anomalies via the proposed process, many of which would have otherwise been missed by traditional means.

[1]  Yoohwan Kim,et al.  Text mining for security threat detection discovering hidden information in unstructured log messages , 2016, 2016 IEEE Conference on Communications and Network Security (CNS).

[2]  Michael J. Chapple,et al.  System Anomaly Detection: Mining Firewall Logs , 2006, 2006 Securecomm and Workshops.

[3]  Trevor J. Bihl,et al.  Cyber-Physical Security with RF Fingerprint Classification through Distance Measure Extensions of Generalized Relevance Learning Vector Quantization , 2020, Secur. Commun. Networks.

[4]  Dorothy E. Denning,et al.  An Intrusion-Detection Model , 1987, IEEE Transactions on Software Engineering.

[5]  Robert G. Abbott,et al.  Log Analysis of Cyber Security Training Exercises , 2015 .

[6]  Wayne G. Lutters,et al.  Supporting intrusion detection work practice , 2009 .

[7]  André Karpistsenko,et al.  Networked Intelligence: Towards Autonomous Cyber Physical Systems , 2016, ArXiv.

[8]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Olivier Festor,et al.  HuMa: A Multi-layer Framework for Threat Analysis in a Heterogeneous Log Environment , 2017, FPS.

[10]  Jakub Breier,et al.  A Dynamic Rule Creation Based Anomaly Detection Method for Identifying Security Breaches in Log Records , 2015, Wireless Personal Communications.

[11]  Rayford B. Vaughn,et al.  Deterministic Intrusion Detection Rules for MODBUS Protocols , 2013, 2013 46th Hawaii International Conference on System Sciences.

[12]  Trevor J. Bihl,et al.  Security Methods for Critical Infrastructure Communications , 2018, Big Data Analytics in Future Power Systems.

[13]  Yao Yuan Chow Application of Data Analytics to Cyber Forensic Data , 2016 .

[14]  Anna Carlin,et al.  Is the Open Way a Better Way? Digital Forensics Using Open Source Tools , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[15]  Mohiuddin Ahmed,et al.  A survey of network anomaly detection techniques , 2016, J. Netw. Comput. Appl..

[16]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[17]  Chun-Hung Richard Lin,et al.  Intrusion detection system: A comprehensive review , 2013, J. Netw. Comput. Appl..

[18]  Gregory White,et al.  A Taxonomy of Cyber Events Affecting Communities , 2011, 2011 44th Hawaii International Conference on System Sciences.

[19]  Trevor J. Bihl,et al.  Cyber anomaly detection: Using tabulated vectors and embedded analytics for efficient data mining , 2018, Journal of Algorithms & Computational Technology.

[20]  Jérôme François,et al.  Topological analysis and visualisation of network monitoring data: Darknet case study , 2016, 2016 IEEE International Workshop on Information Forensics and Security (WIFS).

[21]  Rajdeep Niyogi,et al.  Network forensic frameworks: Survey and research challenges , 2010, Digit. Investig..

[22]  Ludovic Duponchel,et al.  Topological data analysis: A promising big data exploration tool in biology, analytical chemistry and physical chemistry. , 2016, Analytica chimica acta.

[23]  Facundo Mémoli,et al.  Topological Methods for the Analysis of High Dimensional Data Sets and 3D Object Recognition , 2007, PBG@Eurographics.

[24]  William A. Young,et al.  Defining, Understanding, and Addressing Big Data , 2016 .

[25]  Wei Lin,et al.  StreamScope: Continuous Reliable Distributed Processing of Big Data Streams , 2016, NSDI.

[26]  Nadeem Javaid,et al.  Near-miss situation based visual analysis of SIEM rules for real time network security monitoring , 2019, J. Ambient Intell. Humaniz. Comput..

[27]  Mahesan Niranjan,et al.  Improved understanding of aqueous solubility modeling through topological data analysis , 2018, Journal of Cheminformatics.

[28]  Ronald Paans,et al.  A Framework for Designing a Security Operations Centre (SOC) , 2015, 2015 48th Hawaii International Conference on System Sciences.

[29]  Gregory B. White,et al.  Analysis of Payload Based Application level Network Anomaly Detection , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[30]  Mohammad Zulkernine,et al.  Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection , 2006, 2006 IEEE International Conference on Communications.

[31]  Daniel Gmach,et al.  Distributed Real-Time Event Analysis , 2015, 2015 IEEE International Conference on Autonomic Computing.

[32]  Mahdi Zamani,et al.  Machine Learning Techniques for Intrusion Detection , 2013, ArXiv.

[33]  Robert F. Mills,et al.  Design and Analysis of a Dynamically Configured Log-based Distributed Security Event Detection Methodology , 2012 .