Mitigating uncertainty at design time and run time to address assurance for dynamically adaptive systems

MITIGATING UNCERTAINTY AT DESIGN TIME AND RUN TIME TO ADDRESS ASSURANCE FOR DYNAMICALLY ADAPTIVE SYSTEMS By Erik M. Fredericks A dynamically adaptive system (DAS) is a software system that monitors itself and its environment at run time to identify conditions that require self-reconfiguration to ensure that the DAS continually satisfies its requirements. Self-reconfiguration enables a DAS to change its configuration while executing to mitigate unexpected changes. While it is infeasible for an engineer to enumerate all possible conditions that a DAS may experience, the DAS must still deliver acceptable behavior in all situations. This dissertation introduces a suite of techniques that addresses assurance for a DAS in the face of both system and environmental uncertainty at different levels of abstraction. We first present a technique for automatically incorporating flexibility into system requirements for different configurations of environmental conditions. Second, we describe a technique for exploring the code-level impact of uncertainty on a DAS. Third, we discuss a run-time testing feedback loop to continually assess DAS behavior. Lastly, we present two techniques for introducing adaptation into run-time testing activities. We demonstrate these techniques with applications from two different domains: an intelligent robotic vacuuming system that must clean a room safely and efficiently and a remote data mirroring network that must efficiently and effectively disseminate data throughout the network. We also provide an end-to-end example demonstrating the effectiveness of each assurance technique as applied to the remote data mirroring application. Copyright by ERIK M. FREDERICKS 2015 To Natalie and Zoe, thank you for everything. I couldn’t have done this without you.

[1]  John Mylopoulos,et al.  (Requirement) evolution requirements for adaptive systems , 2012, 2012 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[2]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[3]  Koushik Sen,et al.  Automated Systematic Testing of Open Distributed Programs , 2006, FASE.

[4]  Marc Parizeau,et al.  Genericity in Evolutionary Computation Software Tools: Principles and Case-study , 2006, Int. J. Artif. Intell. Tools.

[5]  Phil McMinn,et al.  Search-Based Software Testing: Past, Present and Future , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[6]  Naeem Esfahani,et al.  A framework for managing uncertainty in self-adaptive software systems , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[7]  Peter Sawyer,et al.  Understanding the Scope of Uncertainty in Dynamically Adaptive Systems , 2010, REFSQ.

[8]  Margus Veanes,et al.  Online Testing with Reinforcement Learning , 2006, FATES/RV.

[9]  Rogério de Lemos,et al.  Evaluation of resilience in self-adaptive systems using probabilistic model-checking , 2012, 2012 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[10]  Marsha Chechik,et al.  On the consistency, expressiveness, and precision of partial modeling formalisms , 2011, Inf. Comput..

[11]  Carlo Ghezzi,et al.  Model evolution by run-time parameter adaptation , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[12]  Michael D. Ernst,et al.  An experimental evaluation of continuous testing during development , 2004, ISSTA '04.

[13]  Luciano Baresi,et al.  Validation of web service compositions , 2007, IET Softw..

[14]  Ronald F. DeMara,et al.  Autonomous FPGA fault handling through competitive runtime reconfiguration , 2005, 2005 NASA/DoD Conference on Evolvable Hardware (EH'05).

[15]  Anna Perini,et al.  Automated Continuous Testing of Multi-Agent Systems , 2007 .

[16]  Yuanyuan Zhang,et al.  Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications , 2009 .

[17]  Kalyanmoy Deb,et al.  Approximating a multi-dimensional Pareto front for a land use management problem: A modified MOEA with an epigenetic silencing metaphor , 2012, 2012 IEEE Congress on Evolutionary Computation.

[18]  Barbara G. Ryder,et al.  Constructing the Call Graph of a Program , 1979, IEEE Transactions on Software Engineering.

[19]  K. V. D. Hauw Evaluating and Improving Steady State Evolutionary Algorithms on Constraint Satisfaction Problems , 1996 .

[20]  Nelly Bencomo,et al.  A Goal-Based Modeling Approach to Develop Requirements of an Adaptive System with Environmental Uncertainty , 2009, MoDELS.

[21]  Nelly Bencomo,et al.  Supporting Decision-Making for Self-Adaptive Systems: From Goal Models to Dynamic Decision Networks , 2013, REFSQ.

[22]  Mark Harman,et al.  Search Based Software Engineering: Techniques, Taxonomy, Tutorial , 2010, LASER Summer School.

[23]  Axel van Lamsweerde,et al.  Handling Obstacles in Goal-Oriented Requirements Engineering , 2000, IEEE Trans. Software Eng..

[24]  Jeffrey K. Hollingsworth,et al.  Efficient instrumentation for code coverage testing , 2002, ISSTA '02.

[25]  Dirk Beyer,et al.  Designing for Disasters , 2004, FAST.

[26]  Martin S. Feather,et al.  Requirements monitoring in dynamic environments , 1995, Proceedings of 1995 IEEE International Symposium on Requirements Engineering (RE'95).

[27]  Luciano Baresi,et al.  Fuzzy Goals for Requirements-Driven Adaptation , 2010, 2010 18th IEEE International Requirements Engineering Conference.

[28]  John Wilkes,et al.  Seneca: remote mirroring done write , 2003, USENIX Annual Technical Conference, General Track.

[29]  Jeffrey O. Kephart,et al.  The Vision of Autonomic Computing , 2003, Computer.

[30]  Sungwon Kang,et al.  A Weighted Call Graph Approach for Finding Relevant Components in Source Code , 2009, 2009 10th ACIS International Conference on Software Engineering, Artificial Intelligences, Networking and Parallel/Distributed Computing.

[31]  A. E. Eiben,et al.  Adaptive Penalties for Evolutionary Graph Coloring , 1997, Artificial Evolution.

[32]  Giuseppe Valetto,et al.  Elicitation and utilization of application-level utility functions , 2009, ICAC '09.

[33]  Stephen Fickas,et al.  Goal-Directed Requirements Acquisition , 1993, Sci. Comput. Program..

[34]  Chris Ford,et al.  Non-Functional Requirements , 2007 .

[35]  Bradley R. Schmerl,et al.  Architecture-based self-adaptation in the presence of multiple objectives , 2006, SEAMS '06.

[36]  Daniel P. Siewiorek,et al.  FIAT-fault injection based automated testing environment , 1988, [1988] The Eighteenth International Symposium on Fault-Tolerant Computing. Digest of Papers.

[37]  Jeff Magee,et al.  The Evolving Philosophers Problem: Dynamic Change Management , 1990, IEEE Trans. Software Eng..

[38]  Carlo Ghezzi,et al.  A formal approach to adaptive software: continuous assurance of non-functional requirements , 2011, Formal Aspects of Computing.

[39]  Tim Menzies,et al.  Genetic Algorithms for Randomized Unit Testing , 2011, IEEE Transactions on Software Engineering.

[40]  David S. Rosenblum,et al.  Known unknowns: testing in the presence of uncertainty , 2014, SIGSOFT FSE.

[41]  David B. Knoester,et al.  Applying genetic algorithms to decision making in autonomic computing systems , 2009, ICAC '09.

[42]  Charles Ofria,et al.  Avida , 2004, Artificial Life.

[43]  Farnam Jahanian,et al.  Testing of fault-tolerant and real-time distributed systems via protocol fault injection , 1996, Proceedings of Annual Symposium on Fault Tolerant Computing.

[44]  Brice Morin,et al.  Models@ Run.time to Support Dynamic Adaptation , 2009, Computer.

[45]  Michael R. Lyu,et al.  Effect of code coverage on software reliability measurement , 2001, IEEE Trans. Reliab..

[46]  G. Spanoudakis,et al.  A Framework for Requirements Monitoring of Service Based Systems , 2004 .

[47]  Axel van Lamsweerde,et al.  Requirements Engineering: From System Goals to UML Models to Software Specifications , 2009 .

[48]  Edward J. McCluskey,et al.  Dependable Computing and Online Testing in Adaptive and Configurable Systems , 2000, IEEE Des. Test Comput..

[49]  Richard M. Murray,et al.  Verifying Cyber-Physical Interactions in Safety-Critical Systems , 2013, IEEE Security & Privacy.

[50]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[51]  Anna Perini,et al.  A Goal-Oriented Software Testing Methodology , 2007, AOSE.

[52]  Ilene Burnstein,et al.  Practical Software Testing: A Process-Oriented Approach , 2003 .

[53]  Dietmar P. F. Möller,et al.  Systems and Software Engineering , 2016 .

[54]  Nelly Bencomo,et al.  Towards requirements aware systems: Run-time resolution of design-time assumptions , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[55]  Graham Kendall,et al.  Hyper-Heuristics: An Emerging Direction in Modern Search Technology , 2003, Handbook of Metaheuristics.

[56]  George Spanoudakis,et al.  A framework for requirents monitoring of service based systems , 2004, ICSOC '04.

[57]  Andres J. Ramirez,et al.  Automatic derivation of utility functions for monitoring software requirements , 2011, MODELS'11.

[58]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[59]  Hans-Paul Schwefel,et al.  Numerical Optimization of Computer Models , 1982 .

[60]  Nelly Bencomo,et al.  Requirements-Aware Systems: A Research Agenda for RE for Self-adaptive Systems , 2010, 2010 18th IEEE International Requirements Engineering Conference.

[61]  Paul E. Black,et al.  Dictionary of Algorithms and Data Structures | NIST , 1998 .

[62]  David B. Knoester,et al.  Automatically exploring how uncertainty impacts behavior of dynamically adaptive systems , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[63]  John J. Marciniak,et al.  Encyclopedia of Software Engineering , 1994, Encyclopedia of Software Engineering.

[64]  Anna Perini,et al.  Reasoning about adaptive requirements for self-adaptive systems at runtime , 2011, 2011 2nd International Workshop on Requirements@Run.Time.

[65]  Andres J. Ramirez,et al.  Validating Code-Level Behavior of Dynamic Adaptive Systems in the Face of Uncertainty , 2013, SSBSE.

[66]  Marco Canini,et al.  Fault prediction in distributed systems gone wild , 2010, LADIS '10.

[67]  Rajarshi Das,et al.  Utility functions in autonomic systems , 2004, International Conference on Autonomic Computing, 2004. Proceedings..

[68]  Manuel V. Hermenegildo,et al.  Integrating Software Testing and Run-Time Checking in an Assertion Verification Framework , 2009, ICLP.

[69]  Betty H. C. Cheng,et al.  Model-based development of dynamically adaptive software , 2006, ICSE.

[70]  Axel van Lamsweerde,et al.  Reasoning about partial goal satisfaction for requirements and design engineering , 2004, SIGSOFT '04/FSE-12.

[71]  Marco Canini,et al.  Toward Online Testing of Federated and Heterogeneous Distributed Systems , 2011, USENIX Annual Technical Conference.

[72]  Harlan D. Mills,et al.  Cleanroom Software Engineering , 1987, IEEE Software.

[73]  Peyman Oreizy,et al.  An architecture-based approach to self-adaptive software , 1999, IEEE Intell. Syst..

[74]  Edward J. McCluskey,et al.  Reconfigurable architecture for autonomous self-repair , 2004, IEEE Design & Test of Computers.

[75]  Jonathan A. Bauer,et al.  Test Plan Generation Using Formal Grammars , 1979, ICSE.

[76]  Betty H. C. Cheng,et al.  AutoRELAX: automatically RELAXing a goal model to address uncertainty , 2014, Empirical Software Engineering.

[77]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[78]  S. Neema,et al.  Development Environment for Dynamically Reconfigurable Embedded Systems , 1999 .

[79]  Carlo Ghezzi,et al.  Run-time efficient probabilistic model checking , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[80]  Ladan Tahvildari,et al.  Self-adaptive software: Landscape and research challenges , 2009, TAAS.

[81]  Hausi A. Müller,et al.  A framework for evaluating quality-driven self-adaptive software systems , 2011, SEAMS '11.

[82]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999, Softw. Test. Verification Reliab..

[83]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[84]  Martin Gogolla,et al.  Using Models at Runtime to Address Assurance for Self-Adaptive Systems , 2015, Models@run.time@Dagstuhl.

[85]  Andres J. Ramirez,et al.  Towards run-time testing of dynamic adaptive systems , 2013, 2013 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[86]  Pamela Zave,et al.  Deriving Specifications from Requirements: an Example , 1995, 1995 17th International Conference on Software Engineering.

[87]  Seyed Masoud Sadjadi,et al.  Composing adaptive software , 2004, Computer.

[88]  Heather Goldsby,et al.  Automatically Generating Behavioral Models of Adaptive Systems to Address Uncertainty , 2008, MoDELS.

[89]  Michael D. Ernst,et al.  Reducing wasted development time via continuous testing , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[90]  Kenneth O. Stanley,et al.  Exploiting Open-Endedness to Solve Problems Through the Search for Novelty , 2008, ALIFE.

[91]  Earl T. Barr,et al.  Uncertainty, risk, and information value in software requirements and architecture , 2014, ICSE.

[92]  Sebastián Uchitel,et al.  Automated reliability estimation over partial systematic explorations , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[93]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[94]  Jeffrey J. P. Tsai,et al.  A Noninterference Monitoring and Replay Mechanism for Real-Time Software Testing and Debugging , 1990, IEEE Trans. Software Eng..

[95]  A. E. Eiben,et al.  On-Line, On-Board Evolution of Robot Controllers , 2009, Artificial Evolution.

[96]  Hélène Waeselynck,et al.  STATEMATE applied to statistical software testing , 1993, ISSTA '93.

[97]  Carlo Ghezzi Adaptive Software Needs Continuous Verification , 2010, 2010 8th IEEE International Conference on Software Engineering and Formal Methods.

[98]  Betty H. C. Cheng,et al.  Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty , 2014, SEAMS 2014.

[99]  Jochen Ludewig,et al.  Models in software engineering – an introduction , 2003, Software and Systems Modeling.

[100]  Mary Lou Soffa,et al.  Hierarchical GUI Test Case Generation Using Automated Planning , 2001, IEEE Trans. Software Eng..

[101]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[102]  David Garlan,et al.  Rainbow: architecture-based self-adaptation with reusable infrastructure , 2004 .

[103]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[104]  Betty H. C. Cheng,et al.  Automated Generation of Adaptive Test Plans for Self-Adaptive Systems , 2015, 2015 IEEE/ACM 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems.

[105]  Gail E. Kaiser,et al.  An Approach to Autonomizing Legacy Systems , 2002 .

[106]  Thanh-Hung Nguyen,et al.  Runtime Verification of Component-Based Systems , 2011, SEFM.

[107]  John Mylopoulos,et al.  From awareness requirements to adaptive systems: A control-theoretic approach , 2011, 2011 2nd International Workshop on Requirements@Run.Time.

[108]  Andres J. Ramirez,et al.  Automatically RELAXing a Goal Model to Cope with Uncertainty , 2012, SSBSE.

[109]  Bertrand Meyer,et al.  Seven Principles of Software Testing , 2008, Computer.

[110]  Nelly Bencomo,et al.  Requirements reflection: requirements as runtime entities , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[111]  Nelly Bencomo,et al.  RELAX: Incorporating Uncertainty into the Specification of Self-Adaptive Systems , 2009, 2009 17th IEEE International Requirements Engineering Conference.

[112]  Mary Shaw,et al.  Software Engineering for Self-Adaptive Systems: A Research Roadmap , 2009, Software Engineering for Self-Adaptive Systems.

[113]  Sam Malek,et al.  Taming uncertainty in self-adaptive software , 2011, ESEC/FSE '11.

[114]  Jochen Ludewig,et al.  Models in software engineering - an introduction , 2004, Informatik Forschung und Entwicklung.