Analysis of series of measurements from job-centric monitoring by statistical functions

The rising number of executed programs (jobs) enabled by the growing amount of available resources from Clouds, Grids, and HPC (for example) has resulted in an enormous number of jobs. Nowadays, most of the executed jobs are mainly unobserved, so unusual behavior, non-optimal resource usage, and silent faults are not systematically searched and analyzed. Job-centric monitoring enables permanent job observation and, thus, enables the analysis of monitoring data.  In this paper, we show how statistic functions can be used to analyze job-centric monitoring data and how the methods compare to more-complex analysis methods. Additionally, we present the usefulness of job-centric monitoring based on practical experiences.

[1]  H. Lacker,et al.  A new approach to a global fit of the CKM matrix , 2001, hep-ph/0104062.

[2]  Marcel Baláz,et al.  Genetic method for compressed skewed-load delay test generation , 2012, 2012 IEEE 15th International Symposium on Design and Diagnostics of Electronic Circuits & Systems (DDECS).

[3]  Vern Paxson,et al.  Bro: a system for detecting network intruders in real-time , 1998, Comput. Networks.

[4]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[5]  Daniel Ch. von Grünigen.  Digitale Signalverarbeitung: mit einer Einführung in die kontinuierlichen Signale und Systeme , 2008 .

[6]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[7]  Sam Kwong,et al.  Genetic algorithms and their applications , 1996, IEEE Signal Process. Mag..

[8]  Julie A. Dickerson,et al.  Fuzzy network profiling for intrusion detection , 2000, PeachFuzz 2000. 19th International Conference of the North American Fuzzy Information Processing Society - NAFIPS (Cat. No.00TH8500).

[9]  Torsten Hoefler,et al.  Characterizing the Influence of System Noise on Large-Scale Applications by Simulation , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[10]  Eugene W. Myers,et al.  AnO(ND) difference algorithm and its variations , 1986, Algorithmica.

[11]  Dan Gusfield,et al.  Algorithms on Strings, Trees, and Sequences - Computer Science and Computational Biology , 1997 .

[12]  Salvatore J. Stolfo,et al.  A framework for constructing features and models for intrusion detection systems , 2000, TSEC.

[13]  S. E. Smaha.  Haystack: an intrusion detection system , 1988, [Proceedings 1988] Fourth Aerospace Computer Security Applications.

[14]  Ralph Müller-Pfefferkorn,et al.  Cross-Correlation as Tool to Determine the Similarity of Series of Measurements for Big-Data Analysis Tasks , 2015, CloudCom-Asia.

[15]  R. Bellman.  Dynamic programming. , 1957, Science.

[16]  Matthias Weber,et al.  Automatic Analysis of Large Data Sets: A Walk-Through on Methods from Different Perspectives , 2013, 2013 International Conference on Cloud Computing and Big Data.

[17]  Salvatore J. Stolfo,et al.  Data Mining Approaches for Intrusion Detection , 1998, USENIX Security Symposium.

[18]  S. B. Needleman,et al.  A general method applicable to the search for similarities in the amino acid sequence of two proteins. , 1970, Journal of molecular biology.

[19]  Salvatore J. Stolfo,et al.  Toward parallel and distributed learning by meta-learning , 1993 .

[20]  Salvatore J. Stolfo,et al.  A data mining framework for building intrusion detection models , 1999, Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344).

[21]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[22]  S. Laplace,et al.  CP violation and the CKM matrix: assessing the impact of the asymmetric B factories , 2004, hep-ph/0406184.

[23]  Ralph Müller-Pfefferkorn,et al.  Achieving scalability for job centric monitoring in a distributed infrastructure , 2012, ARCS 2012.

[24]  Dorothy E. Denning,et al.  An Intrusion-Detection Model , 1987, IEEE Transactions on Software Engineering.

[25]  John J. Grefenstette,et al.  Optimization of Control Parameters for Genetic Algorithms , 1986, IEEE Transactions on Systems, Man, and Cybernetics.