Model-driven online capacity management for component-based software systems

Software timing behavior measurements, such as response times, often show high statistical variance. This variance can make the analysis difficult or even threaten the applicability of statistical techniques. This thesis introduces a method for improving the analysis of software response time measurements that show high variance. Our approach can find relations between timing behavior variance and both trace shape information and workload intensity information. This relation is used to provide timing behavior measurements with virtually less variance. This can make timing behavior analysis more robust (e.g., improved confidence and precision) and faster (e.g., less simulation runs and shorter monitoring period). The thesis contributes TracSTA (Trace-Context-Sensitive Timing Behavior Analysis) and WiSTA (Workload-Intensity-Sensitive Timing Behavior Analysis). TracSTA uses trace shape information (i.e., the shape of the control flow corresponding to a software operation execution) and WiSTA uses workload intensity metrics (e.g., the number of concurrent software executions) to create context-specific timing behavior profiles. Both the applicability and effectiveness are evaluated in several case studies and field studies. The evaluation shows a strong relation between timing behavior and the metrics considered by TracSTA and WiSTA. Additionally, a fault localization approach for enterprise software systems is presented as application scenario. It uses the timing behavior data provided by TracSTA and WiSTA for anomaly detection.

[1]  Will Cappelli Magic Quadrant for Application Performance Monitoring , 2010 .

[2]  James R. Larus,et al.  The use of program profiling for software maintenance with applications to the year 2000 problem , 1997, ESEC '97/FSE-5.

[3]  Haixun Wang,et al.  Adaptive system anomaly prediction for large-scale hosting infrastructures , 2010, PODC.

[4]  Armando Fox,et al.  Using statistical monitoring to detect failures in internet services , 2005 .

[5]  J. Meigs,et al.  WHO Technical Report , 1954, The Yale Journal of Biology and Medicine.

[6]  George Candea,et al.  Microreboot - A Technique for Cheap Recovery , 2004, OSDI.

[7]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[8]  Malgorzata Steinder,et al.  The present and future of event correlation: A need for end-to-end service fault localization , 2001 .

[9]  Richard M. Bailey,et al.  Performance and Availability Measurement of the IBM Information Network , 1983, IBM Syst. J..

[10]  Wilhelm Hasselbring,et al.  Self-adaptive software system monitoring for performance anomaly localization , 2011, ICAC '11.

[11]  Julio César López-Hernández,et al.  Stardust: tracking activity in a distributed storage system , 2006, SIGMETRICS '06/Performance '06.

[12]  Wilhelm Hasselbring,et al.  Continuous Monitoring of Software Services: Design and Application of the Kieker Framework , 2009 .

[13]  Armando Fox,et al.  Pinpoint: problem determination in large , 2002 .

[14]  Gregory R. Ganger,et al.  Ironmodel: robust performance models in the wild , 2008, SIGMETRICS '08.

[15]  Tony Printezis Garbage Collection in the Java HotSpot Virtual Machine Gain a better understanding of how garbage collection in the Java HotSpot VM works, and learn to take full advantage of it when designing, developing, and deploying your Java applications. , 1999 .

[16]  Jordi Torres,et al.  Autonomic QoS control in enterprise Grid environments using online simulation , 2009, J. Syst. Softw..

[17]  Roy A. Maxion,et al.  Anomaly detection for diagnosis , 1990, [1990] Digest of Papers. Fault-Tolerant Computing: 20th International Symposium.

[18]  I. G. BONNER CLAPPISON Editor , 1960, The Electric Power Engineering Handbook - Five Volume Set.

[19]  A. Zeller Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[20]  Ming-Yee Lai,et al.  Software Fault Insertion Testing for Fault Tolerance , 1995 .

[21]  Leonard J LaPadula State of the Art in Anomaly Detection and Reaction: An Update , 1999 .

[22]  Samuel Kounev,et al.  S/T/A: Meta-Modeling Run-Time Adaptation in Component-Based System Architectures , 2012, 2012 IEEE Ninth International Conference on e-Business Engineering.

[23]  Jerome A. Rolia,et al.  Automatic generation of a software performance model using an object-oriented prototype , 1995, MASCOTS '95. Proceedings of the Third International Workshop on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems.

[24]  Terence Kelly,et al.  Detecting Performance Anomalies in Global Applications , 2005, WORLDS.

[25]  Neil A. Ernst,et al.  The Journal of Systems and Software , 2013 .

[26]  Petr Tuma,et al.  Repeated results analysis for middleware regression benchmarking , 2005, Perform. Evaluation.

[27]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[28]  Sandeep Kumar,et al.  Classification and detection of computer intrusions , 1996 .

[29]  Haifeng Chen,et al.  Online Tracking of Component Interactions for Failure Detection and Localization in Distributed Systems , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[30]  Randy H. Katz,et al.  A view of cloud computing , 2010, CACM.

[31]  Wilhelm Hasselbring,et al.  Live Visualization of Large Software Landscapes for Ensuring Architecture Conformance , 2014, ECSAW '14.

[32]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[33]  Jeffrey S. Chase,et al.  Correlating Instrumentation Data to System States: A Building Block for Automated Diagnosis and Control , 2004, OSDI.

[34]  Boris Gruschke,et al.  INTEGRATED EVENT MANAGEMENT: EVENT CORRELATION USING DEPENDENCY GRAPHS , 1998 .

[35]  Carl E. Landwehr,et al.  Basic concepts and taxonomy of dependable and secure computing , 2004, IEEE Transactions on Dependable and Secure Computing.

[36]  Xiaoqing Cheng,et al.  Performance, Benchmarking and Sizing in Developing Highly Scalable Enterprise Software , 2008, SIPEW.

[37]  Perry Cheng,et al.  Myths and realities: the performance impact of garbage collection , 2004, SIGMETRICS '04/Performance '04.

[38]  Ada Diaconescu,et al.  Automatic performance management in component based software systems , 2004, International Conference on Autonomic Computing, 2004. Proceedings..

[39]  Thomas Bemmerl,et al.  An Integrated Environment for Programming Distributed Memory Multiprocessors , 1991, EDMCC.

[40]  Matthias Hauswirth,et al.  Vertical profiling: understanding the behavior of object-priented applications , 2004, OOPSLA.

[41]  Virgílio A. F. Almeida,et al.  Capacity Planning for Web Services: Metrics, Models, and Methods , 2001 .

[42]  Tillmann Carlos Bielefeld Online performance anomaly detection for large-scale software systems , 2012 .

[43]  Wei Liu,et al.  AccMon: Automatically Detecting Memory-Related Bugs via Program Counter-Based Invariants , 2004, 37th International Symposium on Microarchitecture (MICRO-37'04).

[44]  Sheng Liang,et al.  Java Virtual Machine Profiler Interface , 2000, IBM Syst. J..

[45]  Recommended Practice for Architectural Description of Software-Intensive Systems , 1999 .

[46]  M. Gribaudo,et al.  2002 , 2001, Cell and Tissue Research.

[47]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[48]  Anne Lohrli Chapman and Hall , 1985 .

[49]  Steffen Becker,et al.  Model-Based performance prediction with the palladio component model , 2007, WOSP '07.

[50]  Priya Narasimhan,et al.  Tiresias: Black-Box Failure Prediction in Distributed Systems , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[51]  Marcos K. Aguilera,et al.  WAP5: black-box performance debugging for wide-area systems , 2006, WWW '06.

[52]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[53]  Malgorzata Steinder,et al.  A survey of fault localization techniques in computer networks , 2004, Sci. Comput. Program..

[54]  Cesare Alippi,et al.  Fault Diagnosis Systems , 2014 .

[55]  P. K. Aditya,et al.  A Grammar Based Fault Classification Scheme and its Application to the Classification of the Errors , 1995 .

[56]  Jake D. Brutlag,et al.  Aberrant Behavior Detection in Time Series for Network Monitoring , 2000, LISA.

[57]  Sebastian G. Elbaum,et al.  Trace anomalies as precursors of field failures: an empirical study , 2007, Empirical Software Engineering.

[58]  Elaine J. Weyuker,et al.  Ensuring stable performance for systems that degrade , 2005, WOSP '05.

[59]  Jaesung Choi,et al.  An alarm correlation and fault identification scheme based on OSI managed object classes , 1999, 1999 IEEE International Conference on Communications (Cat. No. 99CH36311).

[60]  M. Crovella,et al.  Heavy-tailed probability distributions in the World Wide Web , 1998 .

[61]  V. Kreinovich,et al.  How to Define Mean, Variance, etc., for Heavy-Tailed Distributions: A Fractal-Motivated Approach , 2012 .

[62]  Tomas Kalibera,et al.  Reducing performance non-determinism via cache-aware page allocation strategies , 2010, WOSP/SIPEW '10.

[63]  M. Tiffany A Survey of Event Correlation Techniques and Related Topics , 2002 .

[64]  Elfriede Dustin,et al.  Effective software testing : 50 specific ways to improve your testing , 2002 .

[65]  Raj Jain,et al.  The Art of Computer Systems Performance Analysis : Tech-niques for Experimental Design , 1991 .

[66]  Douglas B. Williams,et al.  Characteristic function based estimation of stable distribution parameters , 1998 .

[67]  Fabio Ricciato,et al.  A Distribution-Based Approach to Anomaly Detection and Application to 3G Mobile Traffic , 2009, GLOBECOM 2009 - 2009 IEEE Global Telecommunications Conference.

[68]  Samudra Gupta Pro Apache Log4j , 2005 .

[69]  Sudheendra Hangal,et al.  Tracking down software bugs using automatic anomaly detection , 2002, ICSE '02.

[70]  Edward N. Adams,et al.  Optimizing Preventive Service of Software Products , 1984, IBM J. Res. Dev..

[71]  Amin Vahdat,et al.  Pip: Detecting the Unexpected in Distributed Systems , 2006, NSDI.

[72]  Christopher Krügel,et al.  Anomaly detection of web-based attacks , 2003, CCS '03.

[73]  Jean Arlat,et al.  Architectural Issues in Software Fault Tolerance , 1995 .

[74]  Ho-Won Jung,et al.  Measuring software product quality: a survey of ISO/IEC 9126 , 2004, IEEE Software.

[75]  John D. Musa,et al.  Software reliability - measurement, prediction, application , 1987, McGraw-Hill series in software engineering and technology.

[76]  Andreas Zeller,et al.  Lightweight bug localization with AMPLE , 2005, AADEBUG'05.

[77]  Wilhelm Hasselbring,et al.  Kieker: continuous monitoring and on demand visualization of Java software behavior , 2008, ICSE 2008.

[78]  Brian W. Kernighan,et al.  The Practice of Programming , 1999 .

[79]  Luís Moura Silva Comparing Error Detection Techniques for Web Applications: An Experimental Study , 2008, 2008 Seventh IEEE International Symposium on Network Computing and Applications.

[80]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[81]  Petr Tuma,et al.  Automated detection of performance regressions: the mono experience , 2005, 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems.

[82]  Wilhelm Hasselbring,et al.  Live trace visualization for comprehending large software landscapes: The ExplorViz approach , 2013, 2013 First IEEE Working Conference on Software Visualization (VISSOFT).

[83]  Bradley R. Schmerl,et al.  Rainbow: architecture-based self-adaptation with reusable infrastructure , 2004, International Conference on Autonomic Computing, 2004. Proceedings..

[84]  Hans-Peter Kriegel,et al.  LoOP: local outlier probabilities , 2009, CIKM.

[85]  Rolf Isermann,et al.  Trends in the Application of Model Based Fault Detection and Diagnosis of Technical Processes , 1996 .

[86]  Leana Golubchik,et al.  A Study of Web Services Performance Prediction: A Client's Perspective , 2011, 2011 IEEE 19th Annual International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems.

[87]  Daniel T. Larose,et al.  Discovering Knowledge in Data: An Introduction to Data Mining , 2005 .

[88]  Donald Beaver,et al.  Dapper, a Large-Scale Distributed Systems Tracing Infrastructure , 2010 .

[89]  Simon Giesecke,et al.  Architectural styles for early goal-driven middleware platform selection , 2008, Softwaretechnik-Trends.

[90]  Lars Grunske,et al.  Statistical detection of QoS violations based on CUSUM control charts , 2012, ICPE '12.

[91]  Wilhelm Hasselbring,et al.  Timing behavior anomaly detection in enterprise information systems , 2007, ICEIS.

[92]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[93]  Marc Eisenstadt,et al.  My hairiest bug war stories , 1997, CACM.

[94]  Kishor S. Trivedi,et al.  Performance Assurance via Software Rejuvenation: Monitoring, Statistics and Algorithms , 2006, International Conference on Dependable Systems and Networks (DSN'06).

[95]  A. R. Crathorne,et al.  Economic Control of Quality of Manufactured Product. , 1933 .

[96]  Hermann Kopetz,et al.  Fault tolerance, principles and practice , 1990 .

[97]  Bradley R. Schmerl,et al.  Increasing System Dependability through Architecture-Based Self-Repair , 2002, WADS.

[98]  A. Karimi,et al.  Master‟s thesis , 2011 .

[99]  Piotr Rygielski,et al.  Context Change Detection for Resource Allocation in Service-Oriented Systems , 2011, KES.

[100]  Boris Gruschke A New Approach for Event Correlation based on Dependency Graphs , 1998 .

[101]  Dhabaleswar K. Panda,et al.  DMTracker: finding bugs in large-scale parallel programs by detecting anomaly in data movements , 2007, Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (SC '07).

[102]  Nicholas J. Dingle,et al.  Efficient approximation of response time densities and quantiles in stochastic models , 2004, WOSP '04.

[103]  Amer Diwan,et al.  The DaCapo benchmarks: java benchmarking development and analysis , 2006, OOPSLA '06.

[104]  Qi Zhang,et al.  R-Capriccio: A Capacity Planning and Anomaly Detection Tool for Enterprise Services with Live Workloads , 2007, Middleware.

[105]  Wilhelm Hasselbring,et al.  Trace-Context Sensitive Performance Profiling for Enterprise Software Applications , 2008, SIPEW.

[106]  W. K. George,et al.  University of Illinois at Urbana-Champain , 1997 .

[107]  James R. Larus,et al.  Exploiting hardware performance counters with flow and context sensitive profiling , 1997, PLDI '97.

[108]  Wilhelm Hasselbring,et al.  Workload-intensity-sensitive timing behavior analysis for distributed multi-user software systems , 2010, WOSP/SIPEW '10.

[109]  Eric A. Brewer,et al.  Pinpoint: problem determination in large, dynamic Internet services , 2002, Proceedings International Conference on Dependable Systems and Networks.

[110]  Mor Harchol-Balter Scheduling for Server Farms: Approaches and Open Problems , 2008, SIPEW.

[111]  Vasile Palade,et al.  Computational Intelligence Methodologies in Fault Diagnosis: Review and State of the Art , 2006 .

[112]  R. F. Brown,et al.  PERFORMANCE EVALUATION , 2019, ISO 22301:2019 and business continuity management – Understand how to plan, implement and enhance a business continuity management system (BCMS).

[113]  Bill Hoffman Monitoring, at Your Service , 2005, ACM Queue.

[114]  Jan Waller,et al.  Performance Benchmarking of Application Monitoring Frameworks , 2014, Softwaretechnik-Trends.

[115]  Nina Sophie Marwede,et al.  Automatic Failure Diagnosis based on Timing Behavior Anomaly Correlation in Distributed Java Web Applications , 2008 .

[116]  Alan L. Cox,et al.  Whodunit: transactional profiling for multi-tier applications , 2007, EuroSys '07.

[117]  Jasminka Matevska,et al.  Trace-Context Sensitive Performance Models from Monitoring Data of Software Systems , 2008, ISPASS 2008.

[118]  Haifeng Chen,et al.  Modeling and Tracking of Transaction Flow Dynamics for Fault Detection in Complex Systems , 2006, IEEE Transactions on Dependable and Secure Computing.

[119]  Brian Marick,et al.  A survey of software fault surveys , 1990 .

[120]  Bev Littlewood,et al.  Software reliability and dependability: a roadmap , 2000, ICSE '00.

[121]  John D. Musa,et al.  Software Reliability Engineering: More Reliable Software Faster and Cheaper , 2004 .

[122]  Heiko Koziolek,et al.  Parameter dependencies for reusable performance specifications of software components , 2010 .

[123]  D. Steinberg,et al.  Technometrics , 2008 .

[124]  Connie U. Smith,et al.  New Software Performance AntiPatterns: More Ways to Shoot Yourself in the Foot , 2002, Int. CMG Conference.

[125]  Helen E. Harrison,et al.  Proceedings of the 10th USENIX conference on System administration , 1996 .

[126]  A. Mos COMPAS: adaptive performance monitoring of component-based systems , 2004, ICSE 2004.

[127]  Rui Abreu,et al.  A Survey on Software Fault Localization , 2016, IEEE Transactions on Software Engineering.

[128]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[129]  Robert J. Hall,et al.  Call path profiling , 1992, International Conference on Software Engineering.

[130]  Armando Fox,et al.  Detecting application-level failures in component-based Internet services , 2005, IEEE Transactions on Neural Networks.

[131]  Pablo Molinero-Fernández,et al.  Systems with multiple servers under heavy-tailed workloads , 2005, Perform. Evaluation.

[132]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[133]  Jerome A. Rolia,et al.  The Method of Layers , 1995, IEEE Trans. Software Eng..

[134]  Allen B. Downey The structural cause of file size distributions , 2001, SIGMETRICS '01.

[135]  Ulrich Marquard,et al.  SAP Standard Application Benchmarks - IT Benchmarks with a Business Focus , 2008, SIPEW.

[136]  Christopher Stewart,et al.  Performance modeling and system management for multi-component online services , 2005, NSDI.

[137]  Wilhelm Hasselbring,et al.  Generating Probabilistic and Intensity-Varying Workload for Web-Based Software Systems , 2008, SIPEW.

[138]  Andreas Mielke Elements for response-time statistics in ERP transaction systems , 2006, Perform. Evaluation.

[139]  Jung-Min Park,et al.  An overview of anomaly detection techniques: Existing solutions and latest technological trends , 2007, Comput. Networks.

[140]  Jez Humble,et al.  Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation , 2010 .

[141]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[142]  Marcos K. Aguilera,et al.  Performance debugging for distributed systems of black boxes , 2003, SOSP '03.

[143]  Jean Arlat,et al.  IEEE Transactions on Dependable and Secure Computing , 2006 .

[144]  C. Murray Woodside,et al.  Enhanced Modeling and Solution of Layered Queueing Networks , 2009, IEEE Transactions on Software Engineering.

[145]  André van Hoorn Model-driven online capacity management for component-based software systems , 2014 .

[146]  C. Murray Woodside The Relationship of Performance Models to Data , 2008, SIPEW.

[147]  Alan Fekete,et al.  Design-level performance prediction of component-based applications , 2005, IEEE Transactions on Software Engineering.

[148]  Manish Gupta,et al.  Problem Determination Using Dependency Graphs and Run-Time Behavior Models , 2004, DSOM.

[149]  Jim Gray,et al.  Why Do Computers Stop and What Can Be Done About It? , 1986, Symposium on Reliability in Distributed Software and Database Systems.

[150]  Michael I. Jordan,et al.  Failure diagnosis using decision trees , 2004 .

[151]  Paola Inverardi,et al.  Model-based performance prediction in software development: a survey , 2004, IEEE Transactions on Software Engineering.

[152]  Tom Frotscher,et al.  Architecture-Based Multivariate Anomaly Detection for Software Systems , 2013 .

[153]  David Hovemeyer,et al.  Finding bugs is easy , 2004, SIGP.

[154]  Ulrich Herzog,et al.  Formal Methods for Performance Evaluation , 2002, European Educational Forum: School on Formal Methods and Performance Analysis.

[155]  Stefan Axelsson,et al.  Intrusion Detection Systems: A Survey and Taxonomy , 2002 .

[156]  Zhenmin Li,et al.  PR-Miner: automatically extracting implicit programming rules and detecting violations in large software code , 2005, ESEC/FSE-13.

[157]  Vern Paxson,et al.  Empirically derived analytic models of wide-area TCP connections , 1994, TNET.

[158]  Steffen Becker,et al.  Predicting the Performance of Component-Based Software Architectures with Different Usage Profiles , 2007, QoSA.

[159]  Lauren Wood 技術解説 IEEE Internet Computing , 1999 .

[160]  Wilhelm Hasselbring,et al.  Automatic Failure Diagnosis Support in Distributed Large-Scale Software Systems Based on Timing Behavior Anomaly Correlation , 2009, 2009 13th European Conference on Software Maintenance and Reengineering.

[161]  D. Simon Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches , 2006 .

[162]  Norman E. Fenton,et al.  Quantitative Analysis of Faults and Failures in a Complex Software System , 2000, IEEE Trans. Software Eng..

[163]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[164]  Jerome A. Rolia,et al.  Characterizing the scalability of a large web-based shopping system , 2001, ACM Trans. Internet Techn..

[165]  M. Rohr,et al.  Evaluation of control flow traces in software applications for intrusion detection , 2008, 2008 IEEE International Multitopic Conference.

[166]  James R. Larus,et al.  Efficient path profiling , 1996, Proceedings of the 29th Annual IEEE/ACM International Symposium on Microarchitecture. MICRO 29.

[167]  Heiko Koziolek,et al.  Measuring Performance Metrics: Techniques and Tools , 2005, Dependability Metrics.

[168]  R. Dreisbach,et al.  STANFORD UNIVERSITY. , 1914, Science.

[169]  J. L. Moreno-Rebollo,et al.  Outliers: A Formal Approach , 1990 .

[170]  Samuel Kounev,et al.  Self-adaptive workload classification and forecasting for proactive resource provisioning , 2013, ICPE '13.

[171]  Thilo Focke Performance Monitoring von Middleware-basierten Applikationen , 2006 .

[172]  Ravishankar K. Iyer,et al.  FINE: A Fault Injection and Monitoring Environment for Tracing the UNIX System Behavior under Faults , 1993, IEEE Trans. Software Eng..

[173]  Gregory R. Ganger,et al.  Diagnosing Performance Changes by Comparing Request Flows , 2011, NSDI.

[174]  David Notkin,et al.  An Empirical Study of Java Dynamic Call Graph Extractors , 2001 .

[175]  Elke Achtert,et al.  Visual Evaluation of Outlier Detection Models , 2010, DASFAA.

[176]  Algirdas Avizienis,et al.  Software Fault Tolerance , 1989, IFIP Congress.

[177]  Anup K. Ghosh,et al.  Detecting anomalous and unknown intrusions against programs , 1998, Proceedings 14th Annual Computer Security Applications Conference (Cat. No.98EX217).

[178]  Teerat Pitakrat Hora: Online Failure Prediction Framework for Component-based Software Systems Based on Kieker and Palladio , 2013, KPDAYS.

[179]  Stephanie Forrest,et al.  A sense of self for Unix processes , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[180]  Richard Mortier,et al.  Using Magpie for Request Extraction and Workload Modelling , 2004, OSDI.

[181]  David S. Munro,et al.  In: Software-Practice and Experience , 2000 .

[182]  Robert V. Brill,et al.  Applied Statistics and Probability for Engineers , 2004, Technometrics.

[183]  Acm Sigsoft Proceedings of the 24th International Conference on Software Engineering. ICSE 2002 , 2002, Proceedings of the 24th International Conference on Software Engineering. ICSE 2002.

[184]  Abdelwahab Hamou-Lhadj Techniques to simplify the analysis of execution traces for program comprehension , 2006 .

[185]  Dorina C. Petriu,et al.  The Future of Software Performance Engineering , 2007, Future of Software Engineering (FOSE '07).

[186]  Cristiana Rollino,et al.  [Technology and medicine]. , 2014, Giornale italiano di nefrologia : organo ufficiale della Societa italiana di nefrologia.

[187]  F. E. Grubbs Procedures for Detecting Outlying Observations in Samples , 1969 .

[188]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[189]  Flaviu Cristian,et al.  Atomic Broadcast: From Simple Message Diffusion to Byzantine Agreement , 1995, Inf. Comput..

[190]  Connie U. Smith,et al.  Software Performance AntiPatterns; Common Performance Problems and their Solutions , 2001, Int. CMG Conference.

[191]  Ian Gorton,et al.  Evaluating the Performance of EJB Components , 2003, IEEE Internet Comput..

[192]  J. A. Salvato John wiley & sons. , 1994, Environmental science & technology.

[193]  Michael D. Bond,et al.  Probabilistic calling context , 2007, OOPSLA.

[194]  Jerome A. Rolia,et al.  Trace-Based Load Characterization for Gernerating Performance Software Models , 1999, IEEE Trans. Software Eng..

[195]  Michael R. Lyu,et al.  What is software reliability? , 1994, Proceedings of COMPASS'94 - 1994 IEEE 9th Annual Conference on Computer Assurance.

[196]  Eleazar Eskin,et al.  Anomaly Detection over Noisy Data using Learned Probability Distributions , 2000, ICML.

[197]  KatzelaIrene,et al.  Schemes for fault identification in communication networks , 1995 .

[198]  Konrad Slind,et al.  Monitoring distributed systems , 1987, TOCS.

[199]  Miroslaw Malek,et al.  Proactive fault handling for system availability enhancement , 2005, 19th IEEE International Parallel and Distributed Processing Symposium.

[200]  Ada Diaconescu,et al.  Automating the performance management of component-based enterprise systems through the use of redundancy , 2005, ASE '05.

[201]  Wilhelm Hasselbring,et al.  Kieker: a framework for application performance monitoring and dynamic software analysis , 2012, ICPE '12.

[202]  Peter Buchholz,et al.  A Framework for Simulation Models of Service-Oriented Architectures , 2008, SIPEW.

[203]  H. Cleve,et al.  Locating causes of program failures , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[204]  R. Adler,et al.  A practical guide to heavy tails: statistical techniques and applications , 1998 .

[205]  L. Carvajal,et al.  IEEE Transactions on Software Engineering , 2016 .

[206]  Tao Wang,et al.  Workload-aware anomaly detection for Web applications , 2014, J. Syst. Softw..

[207]  André van Hoorn,et al.  Workload-sensitive Timing Behavior Anomaly Detection in Large Software Systems , 2007 .

[208]  Kishor S. Trivedi,et al.  An empirical investigation of fault types in space mission system software , 2010, 2010 IEEE/IFIP International Conference on Dependable Systems & Networks (DSN).

[209]  Andreas Zeller,et al.  Detecting object usage anomalies , 2007, ESEC-FSE '07.

[210]  Elaine J. Weyuker,et al.  Ensuring system performance for cluster and single server systems , 2007, J. Syst. Softw..

[211]  Tomáš Kalibera Performance in Software Development Cycle: Regression Benchmarking , 2006 .

[212]  Wilhelm Hasselbring,et al.  Ein Vorgehensmodell für Performance-Monitoring von Informationssystemlandschaften , 2006, EAI.

[213]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[214]  A. L. Scherr,et al.  AN ANALYSIS OF TIME-SHARED COMPUTER SYSTEMS , 1965 .

[215]  Rob J Hyndman,et al.  Automatic Time Series Forecasting: The forecast Package for R , 2008 .

[216]  Jens Ehlers,et al.  Self-Adaptive Performance Monitoring for Component-Based Software Systems , 2012, Softwaretechnik-Trends.

[217]  Susan L. Graham,et al.  Gprof: A call graph execution profiler , 1982, SIGPLAN '82.

[218]  Wilhelm Hasselbring,et al.  Instrumentierung zum Monitoring mittels Aspekt-orientierter Programmierung , 2007, Software Engineering.

[219]  Armando Fox,et al.  Capturing, indexing, clustering, and retrieving system history , 2005, SOSP '05.

[220]  Richard Mortier,et al.  Magpie: Online Modelling and Performance-aware Systems , 2003, HotOS.

[221]  Uwe Fink,et al.  Performance Solutions A Practical Guide To Creating Responsive Scalable Software , 2016 .

[222]  Isi Mitrani Simulation techniques for discrete event systems , 1982, Cambridge computer science texts.

[223]  A. Waheed,et al.  A Structured Approach to Instrumentation System Development and Evaluation , 1995, Proceedings of the IEEE/ACM SC95 Conference.

[224]  장훈,et al.  [서평]「Computer Organization and Design, The Hardware/Software Interface」 , 1997 .

[225]  Mor Harchol-Balter Task assignment with unknown duration , 2002, JACM.

[226]  Virgílio A. F. Almeida,et al.  A methodology for workload characterization of E-commerce sites , 1999, EC '99.

[227]  Armando Fox,et al.  Ensembles of models for automated diagnosis of system performance problems , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[228]  Wilhelm Hasselbring,et al.  Software-Betriebs-Leitstände für Unternehmensanwendungslandschaften , 2006, GI Jahrestagung.

[229]  Michael R. Lyu Software Reliability Engineering: A Roadmap , 2007, Future of Software Engineering (FOSE '07).

[230]  R. Lathe Phd by thesis , 1988, Nature.

[231]  J. Kellett London , 1914, The Hospital.

[232]  野村 祐士 International Conference on Communications (ICC'99) , 1999 .

[233]  Lars Grunske,et al.  Increasing Dependability of Component-Based Software Systems by Online Failure Prediction (Short Paper) , 2014, 2014 Tenth European Dependable Computing Conference.

[234]  R. Heilman At your service. , 2005, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[235]  Lieven Eeckhout,et al.  Java performance evaluation through rigorous replay compilation , 2008, OOPSLA.

[236]  Doris L. Carver,et al.  TRANSACTIONS ON SOFTWARE ENGINEERING , 2004 .