RESTful Correlation and Consolidation of Distributed Logging Data in Cloud Environments

Due to the availability of virtualization technologies and related cloud infrastructures, the amount and also the complexity of logging data of systems and services grow steadily. Automated correlation and aggregation techniques are required to support a contemporary processing and interpretation of relevant logging data. In the past, this was achieved using highly centralized logging systems. Based on this fact, the paper introduces a prototype for an automated semantical correlation, aggregation and condensation of logging information. The prototype uses RESTful web services to store and analyze the logging data of distributed logging sources. In this context we will also present the special requirements of handling logging systems in highly dynamic infrastructures like enterprise cloud environments, which provide dynamic systems, services and applications. Keywords—Monitoring; Enterprise Cloud; Web Services; Log Analysis; Log Correlation;

[1]  Claudia Canali,et al.  Automated clustering of VMs for scalable cloud monitoring and management , 2012, SoftCOM 2012, 20th International Conference on Software, Telecommunications and Computer Networks.

[2]  Shinji Kikuchi,et al.  Online failure prediction in cloud datacenters by real-time message pattern learning , 2012, 4th IEEE International Conference on Cloud Computing Technology and Science Proceedings.

[3]  Alexander Clemm,et al.  NETradamus: A forecasting system for system event messages , 2010, 2010 IEEE Network Operations and Management Symposium - NOMS 2010.

[4]  Cesare Pautasso,et al.  Restful web services vs. "big"' web services: making the right architectural decision , 2008, WWW.

[5]  Jeffrey V. Nickerson,et al.  Developing web services choreography standards - the case of REST vs. SOAP , 2005, Decis. Support Syst..

[6]  A. Nur Zincir-Heywood,et al.  Interactive learning of alert signatures in High Performance Cluster system logs , 2012, 2012 IEEE Network Operations and Management Symposium.

[7]  Raffael Marty,et al.  Cloud application logging for forensics , 2011, SAC.

[8]  D. Jayathilake,et al.  Towards structured log analysis , 2012, 2012 Ninth International Conference on Computer Science and Software Engineering (JCSSE).

[9]  Randy H. Katz,et al.  Chukwa: A System for Reliable Large-Scale Log Collection , 2010, LISA.

[10]  Robert F. Mills,et al.  Design and Analysis of a Dynamically Configured Log-based Distributed Security Event Detection Methodology , 2012 .

[11]  Rainer Gerhards,et al.  The Syslog Protocol , 2009, RFC.

[12]  Kenneth E. Nawyn.  A Security Analysis of System Event Logging with Syslog , 2003 .

[13]  Kensuke Fukuda.  On the use of weighted syslog time series for anomaly detection , 2011, 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops.

[14]  Chris Lonvick,et al.  The BSD Syslog Protocol , 2001, RFC.

[15]  Yaohui Jin,et al.  Analysis farm: A cloud-based scalable aggregation and query platform for network log analysis , 2011, 2011 International Conference on Cloud and Service Computing.