NetFlow Monitoring and Cyberattack Detection Using Deep Learning With Ceph

Figuring the network’s hidden abnormal behavior can reduce network vulnerability. This paper presents a detailed architecture in which the collected log data of the network can be processed and analyzed. We process and integrate on-campus network information from every router and store the integrated NetFlow log data. Ceph is used as an open-source distributed storage platform that offers high efficiency, high reliability, scalability, and preliminary preprocessing of raw data with Python, removing redundant areas and unification. In the subanalysis, we discover the anomaly event and absolute flow by three times of standard deviation rule. Keras has been used to classify in-time data collected via a cyber-attack and to construct an automatic identifier template through the Recurring Neural Network (RNN) test. The identification accuracy of the optimization model is around 98% in attack detection. Finally, in the MySQL server, the results of the real-time evaluation can be obtained, and the results of the assessment can be displayed via ECharts.

[1]  Li Li,et al.  Using LSTM and GRU neural network methods for traffic flow prediction , 2016, 2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC).

[2]  Andreas Hotho,et al.  Flow-based Network Traffic Generation using Generative Adversarial Networks , 2018, Comput. Secur..

[3]  Rafal Kozik.  Distributing extreme learning machines with Apache Spark for NetFlow-based malware activity detection , 2018, Pattern Recognit. Lett..

[4]  Michal Choras,et al.  A scalable distributed machine learning approach for attack detection in edge computing environments , 2018, J. Parallel Distributed Comput..

[5]  Chao-Tung Yang,et al.  A Heterogeneous Cloud Storage Platform With Uniform Data Distribution by Software-Defined Storage Technologies , 2019, IEEE Access.

[6]  Yannick Chevalier,et al.  A visual analytics approach for the cyber forensics based on different views of the network traffic , 2018, J. Wirel. Mob. Networks Ubiquitous Comput. Dependable Appl..

[7]  Hongyu Liu,et al.  CNN and RNN based payload classification methods for attack detection , 2019, Knowl. Based Syst..

[8]  Sadie Creese,et al.  Modeling Advanced Persistent Threats to enhance anomaly detection techniques , 2018, J. Wirel. Mob. Networks Ubiquitous Comput. Dependable Appl..

[9]  S.A. Brandt,et al.  CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data , 2006, ACM/IEEE SC 2006 Conference (SC'06).

[10]  Seref Sagiroglu,et al.  Big data analytics for network anomaly detection from netflow data , 2017, 2017 International Conference on Computer Science and Engineering (UBMK).

[11]  Chao-Tung Yang,et al.  On construction of a network log management system using ELK Stack with Ceph , 2019, The Journal of Supercomputing.

[12]  Alejandro Zunino,et al.  An empirical comparison of botnet detection methods , 2014, Comput. Secur..

[13]  Howon Kim,et al.  Long Short Term Memory Recurrent Neural Network Classifier for Intrusion Detection , 2016, 2016 International Conference on Platform Technology and Service (PlatCon).

[14]  Sung-Bae Cho,et al.  Web traffic anomaly detection using C-LSTM neural networks , 2018, Expert Syst. Appl..

[15]  Juan E. Tapiador,et al.  Anomaly detection methods in wired networks: a survey and taxonomy , 2004, Comput. Commun..

[16]  Aiko Pras,et al.  Flow Monitoring Explained: From Packet Capture to Data Analysis With NetFlow and IPFIX , 2014, IEEE Communications Surveys & Tutorials.

[17]  Mounir Ghogho,et al.  Deep learning approach for Network Intrusion Detection in Software Defined Networking , 2016, 2016 International Conference on Wireless Networks and Mobile Communications (WINCOM).

[18]  James Zhang,et al.  Anomaly detection in wide area network meshes using two machine learning algorithms , 2019, Future Gener. Comput. Syst..

[19]  Andrew J. Clark,et al.  Data preprocessing for anomaly based network intrusion detection: A review , 2011, Comput. Secur..

[20]  Tarun Soni,et al.  Network Traffic Prediction Using Recurrent Neural Networks , 2018, 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA).

[21]  Rajiv Ranjan,et al.  Implementation of a real-time network traffic monitoring service with network functions virtualization , 2019, Future Gener. Comput. Syst..