Performance Optimization of Multi-Tenant Software Systems

Multi-tenant software systems are Software-as-a-Service systems in which customers (or tenants) share the same resources. The key characteristics of multi-tenancy are hardware resource sharing, a high degree of configurability and a shared application and database instance. We can deduct from these characteristics that they lead to challenges compared to traditional software schemes. To better understand these challenges, we have come up with a reengineering pattern for transforming an existing single-tenant application into a multi-tenant one. We have done a case study in which we transform a single-tenant research prototype into a multi-tenant version. This case study showed that in a layered application, this transformation could be done in less than 100 lines of code. With a better understanding of the challenges inflicted by multi-tenancy, we have focused on one of these challenges in this thesis, namely performance. Because tenants share resources in multi-tenant applications, it is necessary to optimize these applications on two levels: (1) at the hardware level and (2) at the software level. In this thesis, we present and evaluate methods to analyze and optimize the performance of multi-tenant software on those two levels.

[1]  Marija Mikic-Rakic,et al.  An Extensible Framework for Improving a Distributed Software System's Deployment Architecture , 2012, IEEE Transactions on Software Engineering.

[2]  T Purusothaman,et al.  UTILITY SENTIENT FREQUENT ITEM SET MINING AND ASSOCIATION RULE MINING: A LITERATURE SURVEY AND COMPARATIVE STUDY , 2009 .

[3]  Thomas A. Corbi,et al.  The dawning of the autonomic computing era , 2003, IBM Syst. J..

[4]  K. Tsirogiannis,et al.  Analysis of flow and visibility on triangulated terrains , 2011 .

[5]  Carst Tankink.  Documentation and formal mathematics : web technology meets theorem proving , 2013 .

[6]  Arthur I. Baars,et al.  Embedded Compilers , 2009 .

[7]  Stephanie Kemper,et al.  Modelling and analysis of real-time coordination patterns , 2011 .

[8]  Adriaan Middelkoop,et al.  Inference of Program Properties with Attribute Grammars, Revisited , 2012 .

[9]  Lennart C. L. Kats,et al.  Building Blocks for Language Workbenches , 2011 .

[10]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[11]  Armando Fox,et al.  Ensembles of models for automated diagnosis of system performance problems , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[12]  Ali Mesbah,et al.  Analysis and Testing of Ajax-based Single-page Web Applications , 2009 .

[13]  Marius Adrian Marin,et al.  An Integrated System to Manage Crosscutting Concerns in Source Code , 2008 .

[14]  Henry Hoffmann,et al.  Application heartbeats: a generic interface for specifying program performance and goals in autonomous computing environments , 2010, ICAC '10.

[15]  M. J. de Mol,et al.  Reasoning about functional programs : Sparkle, a proof assistant for Clean , 2009 .

[16]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[17]  van Mj Muck Weerdenburg,et al.  Efficient rewriting techniques , 2009 .

[18]  Cheng-Zhong Xu,et al.  Online Measurement of the Capacity of Multi-Tier Websites Using Hardware Performance Counters , 2008, 2008 The 28th International Conference on Distributed Computing Systems.

[19]  Hendrik Michaël van der Bijl,et al.  On changing models in model-based testing , 2011 .

[20]  Rouven Krebs,et al.  A Qualitative Discussion of Different Approaches for Implementing Multi-Tenant SaaS Offerings , 2011, Software Engineering.

[21]  Haixun Wang,et al.  Adaptive system anomaly prediction for large-scale hosting infrastructures , 2010, PODC.

[22]  Roozbeh Farahbod,et al.  Automated root cause isolation of performance regressions during software development , 2013, ICPE '13.

[23]  Kab Kevin Verbeek.  Algorithms for cartographic visualization , 2012 .

[24]  Ljp Luc Engelen.  From napkin sketches to reliable software , 2012 .

[25]  Jja Jeroen Keiren,et al.  Advanced reduction techniques for model checking , 2013 .

[26]  Harry M. Sneed,et al.  Integrating legacy software into a service oriented architecture , 2006, Conference on Software Maintenance and Reengineering (CSMR'06).

[27]  A Adam Koprowski,et al.  Termination of rewriting and its certification , 2004 .

[28]  Christian Krause,et al.  Reconfigurable Component Connectors , 2011 .

[29]  Frederick Chong,et al.  Multi ‐ Tenant Data Architecture June 2006 , 2016 .

[30]  Mari Antonius Cornelis Dekker,et al.  Flexible Access Control for Dynamic Collaborative Environments , 2009 .

[31]  Madeline Diep,et al.  Profiling deployed software: assessing strategies and testing opportunities , 2005, IEEE Transactions on Software Engineering.

[32]  Mohammad Mahdi Jaghoori,et al.  Time At Your Service: Schedulability Analysis of Real-Time and Distributed Services , 2010 .

[33]  Matthias Hauswirth,et al.  Catch me if you can: performance bug detection in the wild , 2011, OOPSLA '11.

[34]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[35]  Yanbo Han,et al.  Feedback-Control-Based Performance Regulation for Multi-Tenant Applications , 2009, 2009 15th International Conference on Parallel and Distributed Systems.

[36]  L. Lensink,et al.  Applying formal methods in software development , 2013 .

[37]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[38]  Rudolf Berrendorf,et al.  PCL - The Performance Counter Library: A Common Interface to Access Hardware Performance Counters on Microprocessors , 1998 .

[39]  Cor-Paul Bezemer,et al.  Enabling multi-tenancy: An industrial experience report , 2010, 2010 IEEE International Conference on Software Maintenance.

[40]  Johan Pouwelse,et al.  Dispersy Bundle Synchronization , 2013 .

[41]  Richard C. Holt,et al.  Exploring software evolution using spectrographs , 2004, 11th Working Conference on Reverse Engineering.

[42]  Cor-Paul Bezemer,et al.  Improving the Diagnostic Capabilities of a Performance Optimization Approach , 2013 .

[43]  Daniel M. Germán,et al.  The promises and perils of mining git , 2009, 2009 6th IEEE International Working Conference on Mining Software Repositories.

[44]  Dan Roth,et al.  Automated and Adaptive Threshold Setting: Enabling Technology for Autonomy and Self-Management , 2005, Second International Conference on Autonomic Computing (ICAC'05).

[45]  de Ism Ivo Jong.  Integration and test strategies for complex manufacturing machines , 2008 .

[46]  Martin R. Neuhäußer,et al.  Model checking nondeterministic and randomly timed systems , 2010 .

[47]  Ileana Buhan,et al.  Cryptographic keys from noisy data, theory and applications , 2008 .

[48]  Martin Bravenboer,et al.  Exercises in Free Syntax. Syntax Definition, Parsing, and Assimilation of Language Conglomerates , 2003 .

[49]  Vahid Garousi,et al.  Traffic-aware stress testing of distributed real-time systems based on UML models using genetic algorithms , 2008, J. Syst. Softw..

[50]  Michael Jiang,et al.  Monitoring multi-tier clustered systems with invariant metric relationships , 2008, SEAMS '08.

[51]  Flavio D. Garcia.  Formal and Computational Cryptography: Protocols, Hashes and Commitments , 2008 .

[52]  Serap A. Savari,et al.  Comparing and Combining Profiles , 2000, J. Instr. Level Parallelism.

[53]  Taolue Chen.  Clocks, Dice and Processes , 2009 .

[54]  J. K. Berendsen,et al.  Abstraction, prices and probability in model checking timed automata , 2010 .

[55]  Slinger Jansen,et al.  Customization Realization in Multi-tenant Web Applications: Case Studies from the Library Sector , 2010, ICWE.

[56]  M. D. Berg,et al.  Optimal Geometric Data Structures , 2007 .

[57]  Armando Fox,et al.  Capturing, indexing, clustering, and retrieving system history , 2005, SOSP '05.

[58]  Daniela E. Damian,et al.  The promises and perils of mining GitHub , 2009, MSR 2014.

[59]  D. E. Nadales Agut,et al.  A Compositional Interchange Format for Hybrid Systems: Design and Implementation , 2012 .

[60]  Dean Jacobs,et al.  Ruminations on Multi-Tenant Databases , 2007, BTW.

[61]  Ahmed E. Hassan,et al.  Automated detection of performance regressions using statistical process control techniques , 2012, ICPE '12.

[62]  Joung Woo Ryu,et al.  Efficiently Maintaining the Performance of an Ensemble Classifier in Streaming Data , 2012, ICHIT.

[63]  Pascal Durr,et al.  Resource-based Verification for Robust Composition of Aspects , 2008 .

[64]  Cynthia Kop.  Higher Order Termination: Automatable Techniques for Proving Termination of Higher-Order Term Rewriting Systems , 2012 .

[65]  Peter Nobel,et al.  Practical performance models for complex, popular applications , 2010, SIGMETRICS '10.

[66]  A. L. de Groot,et al.  Practical Automaton proofs in PVS , 2000 .

[67]  Tian Jiang,et al.  Discovering, reporting, and fixing performance bugs , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[68]  R. Bakhshi.  Gossiping Models : Formal Analysis of Epidemic Protocols , 2011 .

[69]  Craig D. Weissman,et al.  The design of the force.com multitenant internet application development platform , 2009, SIGMOD Conference.

[70]  Tim K. Cocx,et al.  Metrics and visualisation for crime analysis and genomics , 2005 .

[71]  Alex Potanin,et al.  A study of performance variations in the Mozilla Firefox web browser , 2013 .

[72]  Fpm Frank Stappers.  Bridging formal models : an engineering perspective , 2012 .

[73]  van Mf Marcel Amstel,et al.  Assessing and improving the quality of model transformations , 2012 .

[74]  Gheorghe Paun.  Spiking Neural P Systems: A Tutorial , 2007, Bull. EATCS.

[75]  Dorina C. Petriu,et al.  The Future of Software Performance Engineering , 2007, Future of Software Engineering (FOSE '07).

[76]  Anees Shaikh,et al.  Virtualization-Based Techniques for Enabling Multi-tenant Management Tools , 2007, DSOM.

[77]  Sander van der Burg,et al.  A Reference Architecture for Distributed Software Deployment , 2013 .

[78]  Robert Brijder,et al.  Models of natural computation : gene assembly and membrane systems , 2008 .

[79]  Sooyong Park,et al.  Dynamic Software Product Lines , 2008, Computer.

[80]  Trajce Dimkov,et al.  Alignment of organizational security policies: Theory and Practice , 2012 .

[81]  Tingting Han,et al.  Diagnosis, Synthesis and Analysis of Probabilistic Models , 2009, Ausgezeichnete Informatikdissertationen.

[82]  Lionel C. Briand,et al.  Stress testing real-time systems with genetic algorithms , 2005, GECCO '05.

[83]  Jens R. Calamé,et al.  Testing reactive systems with data: enumerative methods and constraint solving , 2008 .

[84]  M Muhammad Atif,et al.  Formal modeling and verification of distributed failure detectors , 2011 .

[85]  Qi Zhang,et al.  R-Capriccio: A Capacity Planning and Anomaly Detection Tool for Enterprise Services with Live Workloads , 2007, Middleware.

[86]  José Proença,et al.  Synchronous Coordination of Distributed Components , 2011 .

[87]  Georgeta Igna,et al.  Performance analysis of real-time task systems using timed automata , 2013 .

[88]  R.S.S. O'Connor,et al.  Incompleteness & completeness : formalizing logic and analysis in type theory , 2005 .

[89]  Marcel Verhoef,et al.  Modeling and validating distributed embedded real-time control systems , 2009 .

[90]  Srinath Perera,et al.  A Multi-tenant Architecture for Business Process Executions , 2011, 2011 IEEE International Conference on Web Services.

[91]  Leland Wilkinson,et al.  The History of the Cluster Heat Map , 2009 .

[92]  Wei-Tek Tsai,et al.  Towards a scalable and robust multi-tenancy SaaS , 2010, Internetware.

[93]  van den,et al.  Composition and synchronization of real-time components upon one processor , 2013 .

[94]  E Elena Mumford,et al.  Drawing graphs for cartographic applications , 2008 .

[95]  U Uzma Khadim,et al.  Process algebras for hybrid systems : comparison and development , 2008 .

[96]  van Pja Paul Tilburg.  From computability to executability : a process-theoretic view on automata theory , 2011 .

[97]  Thomas Bäck,et al.  Mixed-integer evolution strategies for parameter optimization and their applications to medical image analysis , 2005 .

[98]  Gilbert Hamann,et al.  Automatic Comparison of Load Tests to Support the Performance Analysis of Large Enterprise Systems , 2010, 2010 14th European Conference on Software Maintenance and Reengineering.

[99]  Jasen Markovski,et al.  Real and stochastic time in process algebras for performance evaluation , 2008 .

[100]  Alexander Zeier,et al.  Customizing Enterprise Software as a Service Applications: Back-End Extension in a Multi-tenancy Environment , 2009, ICEIS.

[101]  Marcin Czenko,et al.  TuLiP : reshaping trust management , 2009 .

[102]  Somayeh Malakuti Khah Olun Abadi.  Event composition model: achieving naturalness in runtime enforcement , 2011 .

[103]  Farhad Arbab,et al.  Model Checking of Component Connectors , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[104]  Cor-Paul Bezemer,et al.  Detecting and Analyzing Performance Regressions Using a Spectrum-Based Approach , 2013 .

[105]  Wolter Pieters,et al.  La volonté machinale: understanding the electronic voting controversy , 2008 .

[106]  Diwakar Krishnamurthy,et al.  Web workload generation challenges – an empirical investigation , 2012, Softw. Pract. Exp..

[107]  Ahmed E. Hassan,et al.  Identifying performance deviations in thread pools , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[108]  Yann-Gaël Guéhéneuc,et al.  Using design patterns and constraints to automate the detection and correction of inter-class design defects , 2001, Proceedings 39th International Conference and Exhibition on Technology of Object-Oriented Languages and Systems. TOOLS 39.

[109]  William Pugh,et al.  RUBiS revisited: why J2EE benchmarking is hard , 2004, OOPSLA '04.

[110]  D. Graaf,et al.  Mining semi-structured data, theoretical and experimental aspects of pattern evaluation , 2008 .

[111]  B. Lijnse,et al.  TOP to the rescue. Task-oriented programming for incident response applications , 2005 .

[112]  Ichiro Hasuo,et al.  Tracing Anonymity with Coalgebras , 2008 .

[113]  Ronald Middelkoop,et al.  Capturing and exploiting abstract views of states in OO verification , 2011 .

[114]  Eduardo Zambon,et al.  Abstract Graph Transformation - Theory and Practice , 2013 .

[115]  Dina Hadžiosmanović,et al.  The process matters: cyber security in industrial control systems , 2014 .

[116]  Gerardo Canfora,et al.  A wrapping approach for migrating legacy system interactive functionalities to Service Oriented Architectures , 2008, J. Syst. Softw..

[117]  Saeed Sedghi,et al.  Towards Provably Secure Efficiently Searchable Encryption , 2012 .

[118]  D. Costa.  Formal models for component connectors , 2010 .

[119]  W. Kuijper.  Compositional Synthesis of Safety Controllers , 2012 .

[120]  Yanjing Wang,et al.  Epistemic Modelling and Protocol Dynamics , 2010 .

[121]  Ajay Mohindra,et al.  Resource Calculations with Constraints, and Placement of Tenants and Instances for Multi-tenant SaaS Applications , 2008, ICSOC.

[122]  RH Rudolf Mak,et al.  Design and performance analysis of data-independent stream processing systems , 2008 .

[123]  Lionel Mamane,et al.  Interactive mathematical documents: creation and presentation , 2004 .

[124]  E. Burton Swanson,et al.  The dimensions of maintenance , 1976, ICSE '76.

[125]  H. Hansen.  Coalgebraic Modelling : Applications in Automata theory and Modal logic , 2009 .

[126]  Juan Pablo Sandoval Alcocer.  Tracking down software changes responsible for performance loss , 2012, IWST.

[127]  Mark Timmer,et al.  Efficient modelling, generation and analysis of Markov automata , 2013 .

[128]  Aah Ammar Osaiweran.  Formal development of control software in the medical systems domain , 2012 .

[129]  Petr Tuma,et al.  Performance Regression Unit Testing: A Case Study , 2013, EPEW.

[130]  Scw Bas Ploeger,et al.  Improved verification methods for concurrent systems , 2009 .

[131]  Hasan Sözer,et al.  Architecting Fault-Tolerant Software Systems , 2009 .

[132]  Jan Bosch,et al.  On the notion of variability in software product lines , 2001, Proceedings Working IEEE/IFIP Conference on Software Architecture.

[133]  Damian Dudek,et al.  Measures for Comparing Association Rule Sets , 2010, ICAISC.

[134]  Miguel E. Andrés,et al.  Quantitative Analysis of Information Leakage in Probabilistic and Nondeterministic Systems , 2011, ArXiv.

[135]  Magiel Bruntink,et al.  Renovation of idiomatic crosscutting concerns in embedded systems , 2005 .

[136]  Zef Hemel.  Methods and Techniques for the Design and Implementation of Domain-Specific Languages , 2012 .

[137]  A. Narwade,et al.  Faculty of Electrical Engineering, Mathematics and Computer Science , 2015 .

[138]  Colin Potts,et al.  Software-engineering research revisited , 1993, IEEE Software.

[139]  Alexandru Iosup,et al.  TRIBLER: a social-based peer-to-peer system: Research Articles , 2008 .

[140]  Z Zvezdan Protic,et al.  Configuration management for models : generic methods for model comparison and model co-evolution , 2011 .

[141]  C.P.T. deGouw,et al.  Combining Monitoring with Run-Time Assertion Checking , 2013 .

[142]  M. T. de Berg,et al.  Algorithms for Fat Objects: Decompositions and Applications , 2004 .

[143]  Adrianus Johannus Paulus Jeckmans.  Cryptographically-Enhanced Privacy for Recommender Systems , 2014 .

[144]  Hossein Rahmani,et al.  Analysis of protein-protein interaction networks by means of annotated graph mining algorithms , 2012 .

[145]  Ramesh Karri,et al.  Are hardware performance counters a cost effective way for integrity checking of programs , 2011, STC '11.

[146]  Evgenia Smirni,et al.  Anomaly? application change? or workload change? towards automated detection of application performance anomaly and change , 2008, 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN).

[147]  Gürcan Gülesir,et al.  Evolvable Behavior Specifications Using Context-Sensitive Wildcards , 2008 .

[148]  J. van den Berg,et al.  Reasoning about Java programs in PVS using JML , 2009 .

[149]  Petri K. Laine.  The role of SW architecture in solving fundamental problems in object-oriented development of large embedded SW systems , 2001, Proceedings Working IEEE/IFIP Conference on Software Architecture.

[150]  John Businge,et al.  Co-evolution of the Eclipse SDK Framework and Its Third-Party Plug-Ins , 2013, 2013 17th European Conference on Software Maintenance and Reengineering.

[151]  Emmanuele Zambon,et al.  Towards optimal IT availability planning: methods and tools , 2011 .

[152]  Cor-Paul Bezemer,et al.  Multi-tenant SaaS applications: maintenance dream or nightmare? , 2010, IWPSE-EVOL '10.

[153]  Sand Correa,et al.  Statistical Approaches to Predicting and Diagnosing Performance Problems in Component-Based Distributed Systems: An Experimental Evaluation , 2010, 2010 Fourth IEEE International Conference on Self-Adaptive and Self-Organizing Systems.

[154]  Arie van Deursen,et al.  A Controlled Experiment for Program Comprehension through Trace Visualization , 2011, IEEE Transactions on Software Engineering.

[155]  Gerhard de Koning Gans,et al.  Outsmarting smart cards , 2013 .

[156]  Cor-Paul Bezemer,et al.  Locating performance improvement opportunities in an industrial software-as-a-service application , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[157]  Thomas Kwok,et al.  A Software as a Service with Multi-tenancy Support for an Electronic Contract Management Application , 2008, 2008 IEEE International Conference on Services Computing.

[158]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[159]  C. J. Boogerd,et al.  Focusing Automatic Code Inspections , 2010 .

[160]  Das Amrita,et al.  Mining Association Rules between Sets of Items in Large Databases , 2013 .

[161]  J. Kwisthout,et al.  The Computational Complexity of Probabilistic Networks , 2009 .

[162]  Young-Joo Moon,et al.  Stochastic models for quality of service of component connectors , 2011 .

[163]  A. Rodriguez Yakushev,et al.  Towards Getting Generic Programming Ready for Prime Time , 2009 .

[164]  Hugo Jonker,et al.  Security matters : privacy in voting and fairness in digital exchange , 2009 .

[165]  Sudipto Ghosh,et al.  Mining System Tests to Aid Software Maintenance , 1998, Computer.

[166]  Ying Chen,et al.  SPIN: Service Performance Isolation Infrastructure in Multi-tenancy Environment , 2008, ICSOC.

[167]  Karina R. Olmos Joffré.  Strategies for Context Sensitive Program Transformation , 2009 .

[168]  Jun Wang,et al.  TRIBLER: a social‐based peer‐to‐peer system , 2008, IPTPS.

[169]  Rainer Koschke,et al.  On dynamic feature location , 2005, ASE.

[170]  Brad Chen,et al.  Locating System Problems Using Dynamic Instrumentation , 2010 .

[171]  Cor-Paul Bezemer,et al.  Performance optimization of deployed software-as-a-service applications , 2014, J. Syst. Softw..

[172]  Armando Fox,et al.  Three Research Challenges at the Intersection of Machine Learning, Statistical Induction, and Systems , 2005, HotOS.

[173]  Tim K. Cocx,et al.  Algorithmic tools for data-oriented law enforcement , 2009 .

[174]  Bo Gao,et al.  A Framework for Native Multi-Tenancy Application Development and Management , 2007, The 9th IEEE International Conference on E-Commerce Technology and The 4th IEEE International Conference on Enterprise Computing, E-Commerce and E-Services (CEC-EEE 2007).

[175]  Alexandre Bergel,et al.  Execution profiling blueprints , 2012, Softw. Pract. Exp..

[176]  Maria Luisa Villani,et al.  A framework for QoS-aware binding and re-binding of composite web services , 2008, J. Syst. Softw..

[177]  EM Elena Bortnik,et al.  Formal methods in support of SMC design , 2008 .

[178]  Bo Gao,et al.  A Study and Performance Evaluation of the Multi-Tenant Data Tier Design Patterns for Service Oriented Computing , 2008, 2008 IEEE International Conference on e-Business Engineering.

[179]  Andy Zaidman,et al.  Spectrum-based fault diagnosis for service-oriented software systems , 2012, 2012 Fifth IEEE International Conference on Service-Oriented Computing and Applications (SOCA).

[180]  Jay L. Devore,et al.  A Modern Introduction to Probability and Statistics: Understanding Why and How , 2006 .

[181]  Peter Zoeteweij,et al.  An Evaluation of Similarity Coefficients for Software Fault Localization , 2006, 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC'06).

[182]  Joachim M. Buhmann,et al.  The Balanced Accuracy and Its Posterior Distribution , 2010, 2010 20th International Conference on Pattern Recognition.

[183]  Willy Zwaenepoel,et al.  Performance and scalability of EJB applications , 2002, OOPSLA '02.

[184]  Benny Rochwerger,et al.  Oceano-SLA based management of a computing utility , 2001, 2001 IEEE/IFIP International Symposium on Integrated Network Management Proceedings. Integrated Network Management VII. Integrated Management Strategies for the New Millennium (Cat. No.01EX470).

[185]  Jeffrey S. Chase,et al.  Correlating Instrumentation Data to System States: A Building Block for Automated Diagnosis and Control , 2004, OSDI.

[186]  Frank Leymann,et al.  Variability modeling to support customization and deployment of multi-tenant-aware Software as a Service applications , 2009, 2009 ICSE Workshop on Principles of Engineering Service Oriented Systems.

[187]  P. E. Jackson,et al.  Estimates of distributions of random variables for certain computer communications traffic models , 1969, Symposium on Problems in the Optimization of Data Communications Systems.

[188]  Ian H. Witten,et al.  Data mining - practical machine learning tools and techniques, Second Edition , 2005, The Morgan Kaufmann series in data management systems.

[189]  J. van den Bos,et al.  Gathering evidence: Model-driven software engineering in automated digital forensics , 2014 .

[190]  B. J. Arnoldus,et al.  An illumination of the template enigma : software code generation with templates , 2011 .

[191]  Sander Vermolen,et al.  Software Language Evolution , 2008, 2008 15th Working Conference on Reverse Engineering.

[192]  Gerardo Canfora,et al.  Search-based testing of service level agreements , 2007, GECCO '07.

[193]  Gilbert Hamann,et al.  Automated performance analysis of load tests , 2009, 2009 IEEE International Conference on Software Maintenance.

[194]  de A. Bruin,et al.  Service-oriented discovery of knowledge : foundations, implementations and applications , 2010 .

[195]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[196]  John Mylopoulos,et al.  Non-Functional Requirements in Software Engineering , 2000, International Series in Software Engineering.

[197]  Jack J. Dongarra,et al.  On Using Incremental Profiling for the Performance Analysis of Shared Memory Parallel Applications , 2007, Euro-Par.

[198]  Ying Zou,et al.  Mining Performance Regression Testing Repositories for Automated Performance Analysis , 2010, 2010 10th International Conference on Quality Software.

[199]  Mjm Marcel Roeloffzen.  Kinetic data structures in the black-box model , 2013 .

[200]  Nitu,et al.  Configurability in SaaS (software as a service) applications , 2009, ISEC '09.

[201]  Damiano Bolzoni,et al.  Revisiting Anomaly-based Network Intrusion Detection Systems , 2009 .

[202]  Bas Basten,et al.  Ambiguity Detection for Programming Language Grammars , 2011 .

[203]  Ncwm Niels Braspenning.  Model-based integration and testing of high-tech multi-disciplinary systems , 2008 .

[204]  Ivan S. Zapreev.  Model checking Markov chains : techniques and tools , 2008 .

[205]  Dhp Dirk Gerrits.  Pushing and pulling : computing push plans for disk-shaped robots, and dynamic labelings for moving points , 2013 .

[206]  Brendan Gregg.  Visualizing System Latency , 2010, ACM Queue.

[207]  Slinger Jansen,et al.  Integrated development and maintenance of software products to support efficient updating of customer configurations: a case study in mass market ERP software , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[208]  A. Morali,et al.  IT architecture-based confidentiality risk assessment in networks of organizations , 2011 .

[209]  S. Georgievska.  Probability and Hiding in Concurrent Processes ( thesis abstract ) , 2011 .

[210]  J. Endrullis.  Termination and Productivity , 2010 .

[211]  Mohammed G. Khatib.  MEMS-Based Storage Devices : Integration in Energy-Constrained Mobile Systems , 2009 .

[212]  Lacramioara Astefanoaei,et al.  An executable theory of multi-agent systems refinement , 2011 .

[213]  van Mpwj Michiel Osch.  Automated model-based testing of hybrid systems , 2009 .

[214]  Donald E. Knuth,et al.  An empirical study of FORTRAN programs , 1971, Softw. Pract. Exp..

[215]  Atanas Rountev,et al.  Uncovering performance problems in Java applications with reference propagation profiling , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[216]  Harmen Kastenberg.  Graph-based software specification and verification , 2008 .

[217]  Steven P. Reiss,et al.  Visualizing the Java heap to detect memory problems , 2009, 2009 5th IEEE International Workshop on Visualizing Software for Understanding and Analysis.

[218]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[219]  Qingyang Li,et al.  Adaptive Fault Detection for Testing Tenant Applications in Multi-tenancy SaaS Systems , 2013, 2013 IEEE International Conference on Cloud Engineering (IC2E).

[220]  M. G. van der Horst,et al.  Scalable block processing algorithms , 2008 .