Massive distributed and parallel log analysis for organizational security

Security log analysis is extremely useful for uncovering intrusions and anomalies. However, the sheer volume of log data demands new frameworks and techniques of computing and security. We present a lightweight distributed and parallel security log analysis framework that allows organizations to analyze a massive number of system, network, and transaction logs efficiently and scalably. Different from the general distributed frameworks, e.g., MapReduce, our framework is specifically designed for security log analysis. It features a minimum set of necessary properties, such as dynamic task scheduling for streaming logs. For prototyping, we implement our framework in Amazon cloud environments (EC2 and S3) with a basic analysis application. Our evaluation demonstrates the effectiveness of our design and shows the potential of our cloud-based distributed framework in large-scale log analysis scenarios.

[1]  Boualem Benatallah,et al.  Using Mapreduce to Scale Events Correlation Discovery for Business Processes Mining , 2012, BPM.

[2]  Youngseok Lee,et al.  An Internet traffic analysis method with MapReduce , 2010, 2010 IEEE/IFIP Network Operations and Management Symposium Workshops.

[3]  Ken Yocum,et al.  In-situ MapReduce for Log Processing , 2011, USENIX Annual Technical Conference.

[4]  Radu State,et al.  BotCloud: Detecting botnets using MapReduce , 2011, 2011 IEEE International Workshop on Information Forensics and Security.

[5]  Samuel T. King,et al.  Enriching Intrusion Alerts Through Multi-Host Causality , 2005, NDSS.

[6]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[7]  Wei-Yu Chen,et al.  ICAS: An inter-VM IDS Log Cloud Analysis System , 2011, 2011 IEEE International Conference on Cloud Computing and Intelligence Systems.

[8]  Danfeng Yao,et al.  Data Leak Detection as a Service , 2012, SecureComm.

[9]  Farnam Jahanian,et al.  CloudAV: N-Version Antivirus in the Network Cloud , 2008, USENIX Security Symposium.

[10]  Sungryoul Lee,et al.  Kargus: a highly-scalable software-based intrusion detection system , 2012, CCS.

[11]  William E. Johnston,et al.  The NetLogger methodology for high performance distributed systems performance analysis , 1998, Proceedings. The Seventh International Symposium on High Performance Distributed Computing (Cat. No.98TB100244).

[12]  Przemyslaw Kazienko,et al.  Web Spam Detection Using MapReduce Approach to Collective Classification , 2012, CISIS/ICEUTE/SOCO Special Sessions.

[13]  Sudip Saha,et al.  DNS for Massive-Scale Command and Control , 2013, IEEE Transactions on Dependable and Secure Computing.

[14]  Albert B. Jeng,et al.  Cloud-Based Anti-Malware Solution , 2011 .

[15]  Qiang Ma,et al.  Detecting infection onset with behavior-based policies , 2011, 2011 5th International Conference on Network and System Security.

[16]  Kyuseok Shim,et al.  MapReduce Algorithms for Big Data Analysis , 2012, Proc. VLDB Endow..

[17]  Raffael Marty,et al.  Cloud application logging for forensics , 2011, SAC.

[18]  Ismail Tahrawee.  A Cloud-Based Anti-Malware Solution , 2011 .

[19]  Naren Ramakrishnan,et al.  User Intention-Based Traffic Dependence Analysis for Anomaly Detection , 2012, 2012 IEEE Symposium on Security and Privacy Workshops.

[20]  Hui Li,et al.  Elastic stream cloud (ESC): A stream-oriented cloud computing platform for Rich Internet Application , 2010, 2010 International Conference on High Performance Computing & Simulation.

[21]  Maozhen Li,et al.  A MapReduce based parallel SVM for large scale spam filtering , 2011, 2011 Eighth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).

[22]  Lorenzo Martignoni,et al.  A Framework for Behavior-Based Malware Analysis in the Cloud , 2009, ICISS.

[23]  Michele Colajanni,et al.  A Software Architecture for the Analysis of Large Sets of Data Streams in Cloud Infrastructures , 2011, 2011 IEEE 11th International Conference on Computer and Information Technology.

[24]  George Danezis,et al.  Proceedings of the 2012 ACM conference on Computer and communications security , 2012, CCS 2012.