An Iterative Decision-Making Scheme for Markov Decision Processes and Its Application to Self-adaptive Systems

Software is often governed by and thus adapts to phenomena that occur at runtime. Unlike traditional decision problems, where a decision-making model is determined for reasoning, the adaptation logic of such software is concerned with empirical data and is subject to practical constraints. We present an Iterative Decision-Making Scheme IDMS that infers both point and interval estimates for the undetermined transition probabilities in a Markov Decision Process MDP based on sampled data, and iteratively computes a confidently optimal scheduler from a given finite subset of schedulers. The most important feature of IDMS is the flexibility for adjusting the criterion of confident optimality and the sample size within the iteration, leading to a tradeoff between accuracy, data usage and computational overhead. We apply IDMS to an existing self-adaptation framework Rainbow and conduct a case study using a Rainbow system to demonstrate the flexibility of IDMS.

[1]  Robert Givan,et al.  Bounded Parameter Markov Decision Processes , 1997, ECP.

[2]  Laurent El Ghaoui,et al.  Robust Control of Markov Decision Processes with Uncertain Transition Matrices , 2005, Oper. Res..

[3]  Rogério de Lemos,et al.  Evaluation of resilience in self-adaptive systems using probabilistic model-checking , 2012, 2012 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[4]  Giordano Tamburrelli,et al.  Reliability of Run-Time Quality-of-Service Evaluation Using Parametric Model Checking , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[5]  Ambuj Tewari,et al.  Bounded Parameter Markov Decision Processes with Average Reward Criterion , 2007, COLT.

[6]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[7]  J. Gillis,et al.  Matrix Iterative Analysis , 1961 .

[8]  Marta Z. Kwiatkowska,et al.  Automated Verification Techniques for Probabilistic Systems , 2011, SFM.

[9]  Taolue Chen,et al.  Automatic Verification of Competitive Stochastic Systems , 2012, TACAS.

[10]  Sam Malek,et al.  FUSION: a framework for engineering self-tuning self-adaptive software systems , 2010, FSE '10.

[11]  Sam Malek,et al.  Uncertainty in Self-Adaptive Software Systems , 2010, Software Engineering for Self-Adaptive Systems.

[12]  Carlo Ghezzi,et al.  Model evolution by run-time parameter adaptation , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[13]  Christel Baier,et al.  Principles of model checking , 2008 .

[14]  Bradley R. Schmerl,et al.  Proactive self-adaptation under uncertainty: a probabilistic model checking approach , 2015, ESEC/SIGSOFT FSE.

[15]  Julie A. McCann,et al.  A survey of autonomic computing—degrees, models, and applications , 2008, CSUR.

[16]  Mary Shaw,et al.  Leveraging Resource Prediction for Anticipatory Dynamic Configuration , 2007, First International Conference on Self-Adaptive and Self-Organizing Systems (SASO 2007).

[17]  Carlo Ghezzi,et al.  Formal Verification With Confidence Intervals to Establish Quality of Service Properties of Software Systems , 2016, IEEE Transactions on Reliability.

[18]  David Garlan,et al.  Stochastic game analysis and latency awareness for proactive self-adaptation , 2014, SEAMS 2014.

[19]  Henry Hoffmann,et al.  Automated multi-objective control for self-adaptive software design , 2015, ESEC/SIGSOFT FSE.

[20]  M. Puterman.  Chapter 8 Markov decision processes , 1990 .

[21]  Carlo Ghezzi,et al.  Supporting Self-Adaptation via Quantitative Verification and Sensitivity Analysis at Run Time , 2016, IEEE Transactions on Software Engineering.

[22]  David Garlan,et al.  Rainbow: cost-effective software architecture-based self-adaptation , 2008 .

[23]  Valérie Issarny,et al.  Dynamic decision networks for decision-making in self-adaptive systems: A case study , 2013, 2013 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[24]  Bradley R. Schmerl,et al.  Architecture-based self-adaptation in the presence of multiple objectives , 2006, SEAMS '06.

[25]  Radu Calinescu,et al.  Dynamic QoS Management and Optimization in Service-Based Systems , 2011, IEEE Transactions on Software Engineering.

[26]  Carlo Ghezzi,et al.  Managing non-functional uncertainty via model-driven adaptivity , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[27]  Daniel Kuhn,et al.  Robust Markov Decision Processes , 2013, Math. Oper. Res..

[28]  David Garlan,et al.  Augmenting Architectural Modeling to Cope with Uncertainty , 2007 .

[29]  David Garlan,et al.  Handling Uncertainty in Autonomic Systems , 2007 .

[30]  Christel Baier,et al.  Principles of Model Checking (Representation and Mind Series) , 2008 .

[31]  Sam Malek,et al.  Taming uncertainty in self-adaptive software , 2011, ESEC/FSE '11.

[32]  David S. Rosenblum,et al.  Asymptotic Perturbation Bounds for Probabilistic Model Checking with Empirically Determined Probability Parameters , 2016 .

[33]  Sam Malek,et al.  RESISTing reliability degradation through proactive reconfiguration , 2010, ASE.

[34]  Alberto L. Sangiovanni-Vincentelli,et al.  Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties , 2013, CAV.