Automating experimentation with distributed systems using generative techniques

Engineering distributed systems is a challenging activity. This is partly due to their intrinsic complexity, and partly due to the practical obstacles that developers face when evaluating and adjusting their design and implementation decisions. This thesis addresses the latter aspect by providing a framework to automate experiments. The experiment automation framework is designed in a generic and programmable way to be used with different types of distributed systems for wide-ranging experimental goals. It covers three key steps for each experiment: (1) workload generation, (2) experiment deployment and execution, and (3) post-processing. We designed an approach to workload generation, the simulation-based approach, in which the stimuli of the subject system are modeled by simulating its user behaviors and its execution environment variations. The execution trace of the simulation programs constructs a workload. We automate the next two steps with a model-based generative approach. It is founded on workloads and a suite of configuration models that characterize the distributed system under experimentation, the testbed on which the experiment is to be carried out, and their mappings. The models are used by generative techniques to automate construction of a control system for deploying, executing, and post-processing the specific experiment. We have validated our approaches by performing experiments with a variety of distributed systems on different testbeds to achieve wide-ranging experimental goals. Our experience shows that this framework can be readily applied to different kinds of distributed system architectures and distributed testbeds, and that using it for meaningful experimentation, especially in large-scale network environments, is advantageous.

[1]  Saul Greenberg,et al.  How people revisit web pages: empirical findings and implications for the design of history systems , 1997, Int. J. Hum. Comput. Stud..

[2]  Donald F. Towsley,et al.  Code red worm propagation modeling and analysis , 2002, CCS '02.

[3]  Paul Murray,et al.  SmartFrog: Configuration and Automatic Ignition of Distributed Applications , 2003 .

[4]  David R. Karger,et al.  Chord: A scalable peer-to-peer lookup service for internet applications , 2001, SIGCOMM '01.

[5]  Kun-Chan Lan,et al.  Rapid model parameterization from traffic measurements , 2002, TOMC.

[6]  David E. Culler,et al.  The ganglia distributed monitoring system: design, implementation, and experience , 2004, Parallel Comput..

[7]  David S. Rosenblum,et al.  Design and evaluation of a wide-area event notification service , 2001, TOCS.

[8]  Carey L. Williamson,et al.  ProWGen: a synthetic workload generation tool for simulation evaluation of web proxy caches , 2002, Comput. Networks.

[9]  Srinivasan Seshan,et al.  The effects of wide-area conditions on WWW server performance , 2001, SIGMETRICS '01.

[10]  Alexander Egyed.  Dynamic Deployment of Executing and Simulating Software Components , 2004, Component Deployment.

[11]  Yoichi Shinoda,et al.  Automatic configuration and execution of Internet experiments on an actual node-based testbed , 2005, First International Conference on Testbeds and Research Infrastructures for the DEvelopment of NeTworks and COMmunities.

[12]  Helmut Hlavacs,et al.  Modeling user behavior: a layered approach , 1999, MASCOTS '99. Proceedings of the Seventh International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems.

[13]  Joseph D. Touch,et al.  Application deployment in virtual networks using the X-Bone , 2002, Proceedings DARPA Active Networks Conference and Exposition.

[14]  Sudipto Ghosh,et al.  A test management and software visualization framework for heterogeneous distributed applications , 2001, Proceedings Sixth IEEE International Symposium on High Assurance Systems Engineering. Special Topic: Impact of Networking.

[15]  Mike Hibler,et al.  An integrated experimental environment for distributed systems and networks , 2002, OSDI '02.

[16]  Kurt Maly,et al.  Hierarchical Filtering-based Monitoring System for Large-scale Distributed Applications , 1997 .

[17]  Robbert van Renesse,et al.  Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining , 2003, TOCS.

[18]  David A. Patterson,et al.  Towards a framework for automated robustness evaluation of distributed services , 2022 .

[19]  Ben Laurie,et al.  Apache: The Definitive Guide , 1997 .

[20]  John D. Musa,et al.  Operational profiles in software-reliability engineering , 1993, IEEE Software.

[21]  Aniruddha S. Gokhale,et al.  Preserving distributed systems critical properties: a model-driven approach , 2004, IEEE Software.

[22]  Saurabh Bagchi,et al.  Dependency Analysis in Distributed Systems using Fault Injection: Application to Problem Determination in an e-commerce Environment , 2001, DSOM.

[23]  G. Voelker,et al.  On the scale and performance of cooperative Web proxy caching , 2000, OPSR.

[24]  C. Murray Woodside,et al.  Scalability and performance experiments using synthetic distributed server systems , 1996, Distributed Syst. Eng..

[25]  อนุวัฒน์ ศุภชุติกุล,et al.  การบริหารระบบ (System administration) , 2004 .

[26]  Raymond R. Hill,et al.  Discrete-Event Simulation: A First Course , 2007, J. Simulation.

[27]  C. Murray Woodside,et al.  DECALS: distributed experiment control and logging system , 1995, CASCON.

[28]  Aniruddha S. Gokhale,et al.  CCMPerf: A Benchmarking Tool for CORBA Component Model Implementations , 2004, Proceedings. RTAS 2004. 10th IEEE Real-Time and Embedded Technology and Applications Symposium, 2004..

[29]  Mark Carson,et al.  NIST Net: a Linux-based network emulation tool , 2003, CCRV.

[30]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[31]  David E. Culler,et al.  A blueprint for introducing disruptive technology into the Internet , 2003, CCRV.

[32]  John C. Grundy,et al.  Generation of distributed system test-beds from high-level software architecture descriptions , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[33]  Aniruddha S. Gokhale,et al.  Model-driven techniques for evaluating the QoS of middleware configurations for DRE systems , 2005, 11th IEEE Real Time and Embedded Technology and Applications Symposium.

[34]  Robert Adams.  Take command: The m4 macro package , 2002 .

[35]  Paul Barford,et al.  Generating representative Web workloads for network and server performance evaluation , 1998, SIGMETRICS '98/PERFORMANCE '98.

[36]  Krishna P. Gummadi,et al.  Measurement, modeling, and analysis of a peer-to-peer file-sharing workload , 2003, SOSP '03.

[37]  David R. Karger,et al.  Wide-area cooperative storage with CFS , 2001, SOSP.

[38]  Luigi Rizzo,et al.  Dummynet: a simple approach to the evaluation of network protocols , 1997, CCRV.

[39]  Maria Ebling,et al.  SynRGen: an extensible file reference generator , 1994, SIGMETRICS.

[40]  Ed Bailey,et al.  Maximum RPM , 1997 .

[41]  Robert Tappan Morris,et al.  Designing a DHT for Low Latency and High Throughput , 2004, NSDI.

[42]  JklB Qmd.  Workload Generation for ns Simulations of Wide Area Networks And The Internet , 2000 .

[43]  William H. Sanders,et al.  Loki: a state-driven fault injector for distributed systems , 2000, Proceeding International Conference on Dependable Systems and Networks. DSN 2000.

[44]  Ian Clarke,et al.  Protecting Free Expression Online with Freenet , 2002, IEEE Internet Comput..

[45]  Raymond A. Paul,et al.  Coyote: an XML-based framework for Web services testing , 2002, 7th IEEE International Symposium on High Assurance Systems Engineering, 2002. Proceedings..

[46]  Alexander L. Wolf,et al.  Design and Evaluation of a Support Service for Mobile, Wireless Publish/Subscribe Applications , 2003, IEEE Trans. Software Eng..

[47]  Péter Urbán,et al.  Neko: a single environment to simulate and prototype distributed algorithms , 2001, Proceedings 15th International Conference on Information Networking.

[48]  Douglas C. Schmidt,et al.  Skoll: distributed continuous quality assurance , 2004, Proceedings. 26th International Conference on Software Engineering.

[49]  G. CN5MOP946Q,et al.  Characterizing user behavior and network performance in a public wireless lan , .

[50]  Richard S. Hall,et al.  A cooperative approach to support software deployment using the Software Dock , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[51]  Krzysztof Czarnecki,et al.  Generative programming - methods, tools and applications , 2000 .

[52]  András Faragó.  A Graph Theoretic Model for Complex Network Failure Scenarios , 2006 .

[53]  Brent N. Chun.  DART: Distributed Automated Regression Testing for Large-Scale Network Applications , 2004, OPODIS.

[54]  Ben Laurie,et al.  Apache: The Definitive Guide, 3rd Edition , 2005 .

[55]  Richard S. Hall,et al.  An architecture for post-development configuration management in a wide-area network , 1997, Proceedings of 17th International Conference on Distributed Computing Systems.