On construction of a network log management system using ELK Stack with Ceph

A log management system is essential for the networks administrator. With a log management tool, we can collect, store, analyze, archive, and finally dispose of the log information. In this paper, we propose the architecture model of a log management system using ELK Stack with Ceph to provide a safe network, good Wi-Fi signal strength, and adequate backup data mechanism. In this case, we use our campus data of Wi-Fi log and NetFlow log. First, we collect and store data of our Wi-Fi log using Filebeats tool, and then, we use Elasticsearch, Logstash, and Kibana Stack to visualize the Wi-Fi log data. Second, we collect and store our NetFlow log using NFDUMP, and then, we also use ELK Stack to visualize the NetFlow log data. Third, we integrate the Wi-Fi log and NetFlow log data in one architecture using a distributed storage Ceph file system (CephFS). Moreover, we also compare the performance of RADOS Gateway and CephFS for better storage mechanism.

[1]  Clinton Gormley,et al.  Elasticsearch: The Definitive Guide , 2015 .

[2]  Van Nam Nguyen,et al.  An Efficient Log Management System , 2016 .

[3]  Chao-Tung Yang,et al.  Implementation of Ceph Storage with Big Data for Performance Comparison , 2017, ICISA.

[4]  Ankit Singh,et al.  Design and development of generic web based framework for log analysis , 2016, 2016 IEEE Region 10 Conference (TENCON).

[5]  Sokratis K. Katsikas,et al.  Design of a dynamic log management infrastructure using risk and affiliation network analysis , 2018, PCI.

[6]  Ke Zhan,et al.  Optimization of Ceph Reads/Writes Based on Multi-threaded Algorithms , 2016, 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS).

[7]  Patricia Sullivan,et al.  Education for a Smarter Planet: The Future of Learning , 2009 .

[8]  Michael W. Godfrey,et al.  Mining modern repositories with elasticsearch , 2014, MSR 2014.

[9]  D. Menascé,et al.  Automatic Workload Characterization Using System Log Analysis , 2015 .

[10]  Muzameel Ahmed,et al.  Analysis of Logs by Using Logstash , 2016, FICTA.

[11]  Jun Zhang,et al.  A Multi-dimension Measurement Study of a Large Scale Campus WiFi Network , 2018, 2018 IEEE 43rd Conference on Local Computer Networks (LCN).

[12]  Chris Phillips,et al.  Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management , 2012 .

[13]  Deepak Rao,et al.  Data Visualization on GitHub Repository Parameters Using Elastic Search and Kibana , 2018, 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI).

[14]  Vijay Singh Rathore,et al.  Analysis and Comparative Exploration of Elastic Search, MongoDB and Hadoop Big Data Processing , 2018 .

[15]  Chao-Tung Yang,et al.  Implementation of an Intelligent Indoor Environmental Monitoring and management system in cloud , 2019, Future Gener. Comput. Syst..

[16]  Misha Kakkar,et al.  Geo-identification of web users through logs using ELK stack , 2016, 2016 6th International Conference - Cloud System and Big Data Engineering (Confluence).

[17]  Chengliang Liu,et al.  The Study of Mixed Storage Scheme of Private Cloud Platform Based on Ceph , 2018 .

[18]  Irfan-Ullah Awan,et al.  Cyber Threat Intelligence from Honeypot Data Using Elasticsearch , 2018, 2018 IEEE 32nd International Conference on Advanced Information Networking and Applications (AINA).