Analysis of Series of Measurements from Job-Centric Monitoring by Statistical Functions

The rising number of executed programs (jobs) enabled by the growing amount of available resources from Clouds, Grids, and HPC (for example) has resulted in an enormous number of jobs. Nowadays, most of the executed jobs are mainly unobserved, so unusual behavior, non-optimal resource usage, and silent faults are not systematically searched and analyzed. Job-centric monitoring enables permanent job observation and, thus, enables the analysis of monitoring data. In this paper, we show how statistic functions can be used to analyze job-centric monitoring data and how the methods compare to more-complex analysis methods. Additionally, we present the usefulness of job-centric monitoring based on practical experiences.

[1]  Marcel Baláz,et al.  Genetic method for compressed skewed-load delay test generation , 2012, 2012 IEEE 15th International Symposium on Design and Diagnostics of Electronic Circuits & Systems (DDECS).

[2]  Julie A. Dickerson,et al.  Fuzzy network profiling for intrusion detection , 2000, PeachFuzz 2000. 19th International Conference of the North American Fuzzy Information Processing Society - NAFIPS (Cat. No.00TH8500).

[3]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[4]  Roland Wismüller,et al.  Job monitoring and steering in D-Grid's High Energy Physics Community Grid , 2009, Future Gener. Comput. Syst..

[5]  Torsten Hoefler,et al.  Characterizing the Influence of System Noise on Large-Scale Applications by Simulation , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[6]  Sam Kwong,et al.  Genetic algorithms and their applications , 1996, IEEE Signal Process. Mag..

[7]  S. B. Needleman,et al.  A general method applicable to the search for similarities in the amino acid sequence of two proteins. , 1970, Journal of molecular biology.

[8]  Dan Gusfield.  Algorithms on Stings, Trees, and Sequences: Computer Science and Computational Biology , 1997, SIGACT News.

[9]  Vern Paxson,et al.  Bro: a system for detecting network intruders in real-time , 1998, Comput. Networks.

[10]  Dan Boneh,et al.  On genetic algorithms , 1995, COLT '95.

[11]  Salvatore J. Stolfo,et al.  Toward parallel and distributed learning by meta-learning , 1993 .

[12]  John J. Grefenstette,et al.  Optimization of Control Parameters for Genetic Algorithms , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[13]  Ralph Müller-Pfefferkorn,et al.  Cross-Correlation as Tool to Determine the Similarity of Series of Measurements for Big-Data Analysis Tasks , 2015, CloudCom-Asia.

[14]  S. E. Smaha.  Haystack: an intrusion detection system , 1988, [Proceedings 1988] Fourth Aerospace Computer Security Applications.

[15]  Salvatore J. Stolfo,et al.  A data mining framework for building intrusion detection models , 1999, Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344).

[16]  Daniel Ch. von Grünigen.  Digitale Signalverarbeitung: mit einer Einführung in die kontinuierlichen Signale und Systeme , 2008 .

[17]  Salvatore J. Stolfo,et al.  A framework for constructing features and models for intrusion detection systems , 2000, TSEC.

[18]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[19]  Marcus Hilbrich.  Jobzentrisches Monitoring in Verteilten Heterogenen Umgebungen mit Hilfe Innovativer Skalierbarer Methoden , 2014 .

[20]  S. Laplace,et al.  CP violation and the CKM matrix: assessing the impact of the asymmetric B factories , 2004, hep-ph/0406184.

[21]  Salvatore J. Stolfo,et al.  Data Mining Approaches for Intrusion Detection , 1998, USENIX Security Symposium.

[22]  Dorothy E. Denning,et al.  An Intrusion-Detection Model , 1986, 1986 IEEE Symposium on Security and Privacy.

[23]  Ralph Müller-Pfefferkorn,et al.  AMon - a User-Friendly Job Monitoring for the Grid , 2007, CoreGRID.

[24]  Matthias Weber,et al.  Automatic Analysis of Large Data Sets: A Walk-Through on Methods from Different Perspectives , 2013, 2013 International Conference on Cloud Computing and Big Data.

[25]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[26]  Ralph Müller-Pfefferkorn,et al.  Achieving scalability for job centric monitoring in a distributed infrastructure , 2012, ARCS 2012.