Model-based runtime analysis of distributed reactive systems

Reactive distributed systems have pervaded everyday life and objects, but often lack measures to ensure adequate behaviour in the presence of unforeseen events or even errors at runtime. As interactions and dependencies within distributed systems increase, the problem of detecting failures which depend on the exact situation and environment conditions they occur in grows. As a result, not only the detection of failures is increasingly difficult, but also the differentiation between the symptoms of a fault, and the actual fault itself, i.e., the cause of a problem. In this paper, we present a novel and efficient approach for analysing reactive distributed systems at runtime, in that we provide a framework for detecting failures as well as identifying their causes. Our approach is based upon monitoring safety-properties, specified in the linear time temporal logic LTL (respectively, TLTL) to automatically generate monitor components which detect violations of these properties. Based on the results of the monitors, a dedicated diagnosis is then performed in order to identify explanations for the misbehaviour of a system. These may be used to store detailed log files, or to trigger recovery measures. Our framework is built modular, layered, and uses merely a minimal communication overhead - especially when compared to other, similar approaches. Further, we sketch first experimental results from our implementations, and describe how it can be used to build a variety of distributed systems using our techniques.

[1]  Daniel Sheridan Using Fixpoint Characterisations of LTL for Bounded Model Checking , 2002 .

[2]  Edmund M. Clarke,et al.  Characterizing Correctness Properties of Parallel Programs Using Fixpoints , 1980, ICALP.

[3]  Wang Yi,et al.  Timed Automata: Semantics, Algorithms and Tools , 2003, Lectures on Concurrency and Petri Nets.

[4]  Deepak D'Souza A Logical Characterisation of Event Clock Automata , 2003, Int. J. Found. Comput. Sci..

[5]  Henrik Reif Andersen,et al.  Runtime Verification of Timed LTL using Disjunctive Normalized Equation Systems , 2003, RV@CAV.

[6]  Paul Procter,et al.  Longman Dictionary of Contemporary English , 1978 .

[7]  Philippe Roussel,et al.  The birth of Prolog , 1993, HOPL-II.

[8]  David Poole,et al.  A Logical Framework for Default Reasoning , 1988, Artif. Intell..

[9]  Carsten Fritz,et al.  Constructing Büchi Automata from Linear Temporal Logic Using Simulation Relations for Alternating Büchi Automata , 2003, CIAA.

[10]  Orna Kupferman,et al.  Resets vs. Aborts in Linear Temporal Logic , 2003, TACAS.

[11]  Fiona A. C. Polack,et al.  Comparison of Formalisation Approaches of UML Class Constructs in Z and Object-Z , 2003, ZB.

[12]  Joseph Y. Halpern,et al.  Decision procedures and expressiveness in the temporal logic of branching time , 1982, STOC '82.

[13]  Stephen A. Cook,et al.  The complexity of theorem-proving procedures , 1971, STOC.

[14]  Wolfgang Thomas,et al.  Automata on Infinite Objects , 1991, Handbook of Theoretical Computer Science, Volume B: Formal Models and Sematics.

[15]  Peter Struss,et al.  G+DE - The Generalized Diagnosis Engine , 2001 .

[16]  Marc Geilen,et al.  On the Construction of Monitors for Temporal Logic Properties , 2001, RV@CAV.

[17]  Ivar Jacobson,et al.  The Unified Modeling Language User Guide , 1998, J. Database Manag..

[18]  Narciso Martí-Oliet,et al.  The Maude 2.0 System , 2003, RTA.

[19]  Peter Struss,et al.  Automated Diagnosis of Car-Subsystems Based on Qualitative Models , 1997, XPS.

[20]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems, Advanced Lectures [The volume is the outcome of a research seminar that was held in Schloss Dagstuhl in January 2004] , 2005, Model-Based Testing of Reactive Systems.

[21]  Salil P. Vadhan,et al.  The Complexity of Counting in Sparse, Regular, and Planar Graphs , 2002, SIAM J. Comput..

[22]  Wang Yi,et al.  UPPAAL Implementation Secrets , 2002, FTRTFT.

[23]  Pascal Raymond,et al.  The synchronous data flow programming language LUSTRE , 1991, Proc. IEEE.

[24]  George S. Avrunin,et al.  Patterns in property specifications for finite-state verification , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[25]  Bernhard Rumpe,et al.  The UML as a formal modeling notation , 1998, Comput. Stand. Interfaces.

[26]  S. Safra,et al.  On the complexity of omega -automata , 1988, [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science.

[27]  Amir Pnueli,et al.  On the Development of Reactive Systems , 1989, Logics and Models of Concurrent Systems.

[28]  Andreas Zeller,et al.  Why Programs Fail: A Guide to Systematic Debugging , 2005 .

[29]  David Powell,et al.  Failure mode assumptions and assumption coverage , 1992, [1992] Digest of Papers. FTCS-22: The Twenty-Second International Symposium on Fault-Tolerant Computing.

[30]  Mattias Krysander,et al.  Combining AI, FDI, and Statistical Hypothesis-Testing in a Framework for Diagnosis , 2003 .

[31]  David Poole,et al.  Representing diagnosis knowledge , 1994, Annals of Mathematics and Artificial Intelligence.

[32]  Olivier Finkel,et al.  Topology and ambiguity in ω-context free languages , 2003 .

[33]  Manfred Broy,et al.  Specification and development of interactive systems: focus on streams, interfaces, and refinement , 2001 .

[34]  Manfred Broy Software technology - formal methods and scientific foundations , 1999, Inf. Softw. Technol..

[35]  Hantao Zhang,et al.  SATO: An Efficient Propositional Prover , 1997, CADE.

[36]  A. Turing On Computable Numbers, with an Application to the Entscheidungsproblem. , 1937 .

[37]  Stephen A. Edwards,et al.  Design of embedded systems: formal models, validation, and synthesis , 1997, Proc. IEEE.

[38]  Ken Thompson,et al.  Programming Techniques: Regular expression search algorithm , 1968, Commun. ACM.

[39]  Grigore Rosu,et al.  Efficient monitoring of safety properties , 2004, International Journal on Software Tools for Technology Transfer.

[40]  Dan Roth,et al.  On the Hardness of Approximate Reasoning , 1993, IJCAI.

[41]  Philippe Schnoebelen,et al.  The Complexity of Temporal Logic Model Checking , 2002, Advances in Modal Logic.

[42]  Harald Heinecke Automotive System Design - Challenges and Potential , 2005, DATE.

[43]  Stéphane Lafortune,et al.  Failure diagnosis using discrete event models , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[44]  Manfred Broy Requirements Engineering for Embedded Systems) , 2003 .

[45]  Bowen Alpern,et al.  Defining Liveness , 1984, Inf. Process. Lett..

[46]  Stephen A. Edwards,et al.  The synchronous languages 12 years later , 2003, Proc. IEEE.

[47]  Raymond Reiter,et al.  A Theory of Diagnosis from First Principles , 1986, Artif. Intell..

[48]  Dana Fisman,et al.  Reasoning with Temporal Logic on Truncated Paths , 2003, CAV.

[49]  Carlos Delgado Kloos,et al.  Practical Formal Methods for Hardware Design , 2001, Research Reports Esprit.

[50]  Dejan Nickovic,et al.  Real Time Temporal Logic: Past, Present, Future , 2005, FORMATS.

[51]  Peter A. Flach Simply logical - intelligent reasoning by example , 1994, Wiley professional computing.

[52]  Leslie Lamport,et al.  What Good is Temporal Logic? , 1983, IFIP Congress.

[53]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching Time Temporal Logic , 2008, 25 Years of Model Checking.

[54]  Clifton A. Ericson,et al.  Hazard Analysis Techniques for System Safety , 2005 .

[55]  David G. Mitchell,et al.  Finding hard instances of the satisfiability problem: A survey , 1996, Satisfiability Problem: Theory and Applications.

[56]  Amir Pnueli,et al.  Applications of Temporal Logic to the Specification and Verification of Reactive Systems: A Survey of Current Trends , 1986, Current Trends in Concurrency.

[57]  Marcelo d'Amorim,et al.  Efficient Monitoring of omega-Languages , 2005, CAV.

[58]  Patricia Bouyer,et al.  On the Expressiveness of TPTL and MTL , 2005, FSTTCS.

[59]  Bengt Jonsson,et al.  Generating online test oracles from temporal logic specifications , 2003, International Journal on Software Tools for Technology Transfer.

[60]  Jason Lee,et al.  Dynamic monitoring of high-performance distributed applications , 2002, Proceedings 11th IEEE International Symposium on High Performance Distributed Computing.

[61]  Peter Baumgartner,et al.  Tableaux for Diagnosis Applications , 1997, TABLEAUX.

[62]  Jan Romberg,et al.  Loose synchronization of event-triggered networks for distribution of synchronous programs , 2004, EMSOFT '04.

[63]  T. Capers Jones,et al.  Estimating software costs , 1998 .

[64]  Grigore Rosu,et al.  Synthesizing Monitors for Safety Properties , 2002, TACAS.

[65]  Paul Strooper Proceedings of the 2005 Australian Software Engineering Conference , 2004 .

[66]  Georg Gottlob,et al.  Identifying the Minimal Transversals of a Hypergraph and Related Problems , 1995, SIAM J. Comput..

[67]  G.E. Moore,et al.  Cramming More Components Onto Integrated Circuits , 1998, Proceedings of the IEEE.

[68]  Brian C. Williams,et al.  Diagnosing Multiple Faults , 1987, Artif. Intell..

[69]  X YingGuoPeiShengJiaoYuChuBanYou Longman Dictionary of Contemporary English , 1991 .

[70]  Clifton A. Ericson,et al.  Hazard Analysis Techniques for System Safety: Ericson/Hazard Analysis Techniques for System Safety , 2005 .

[71]  Marco Roveri,et al.  Bounded Verification of Past LTL , 2004, FMCAD.

[72]  Brian Randell,et al.  Fundamental Concepts of Dependability , 2000 .

[73]  Brian C. Williams,et al.  Diagnosis with Behavioral Modes , 1989, IJCAI.

[74]  Bell Telephone,et al.  Regular Expression Search Algorithm , 1968 .

[75]  Johan de Kleer,et al.  Fundamentals of model-based diagnosis , 2003 .

[76]  Martin Fowler,et al.  Planning Extreme Programming , 2000 .

[77]  Liam P. Sarsfield,et al.  Safety in the Skies , 2000 .

[78]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[79]  Leslie G. Valiant,et al.  The Complexity of Computing the Permanent , 1979, Theor. Comput. Sci..

[80]  守屋 悦朗,et al.  J.E.Hopcroft, J.D. Ullman 著, "Introduction to Automata Theory, Languages, and Computation", Addison-Wesley, A5変形版, X+418, \6,670, 1979 , 1980 .

[81]  Christoph Weidenbach,et al.  Computing Small Clause Normal Forms , 2001, Handbook of Automated Reasoning.

[82]  Hilary Putnam,et al.  A Computing Procedure for Quantification Theory , 1960, JACM.

[83]  Georg Gottlob,et al.  Hypergraph Transversal Computation and Related Problems in Logic and AI , 2002, JELIA.

[84]  Peter G. Neumann,et al.  Computer-related risks , 1994 .

[85]  Thomas A. Henzinger,et al.  Logics and Models of Real Time: A Survey , 1991, REX Workshop.

[86]  Bowen Alpern,et al.  Recognizing safety and liveness , 2005, Distributed Computing.

[87]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[88]  Robert A. Kowalski,et al.  Semantic Trees in Automatic Theorem-Proving , 1983 .

[89]  Joško Deur,et al.  Automatic Tuning of Electronic Throttle Control Strategy , 2003 .

[90]  M.M. Lehman,et al.  The role and impact of assumptions in software development, maintenance and evolution , 2005, IEEE International Workshop on Software Evolvability (Software-Evolvability'05).

[91]  Patricia Bouyer,et al.  Fault Diagnosis Using Timed Automata , 2005, FoSSaCS.

[92]  Claude Berge,et al.  Hypergraphs - combinatorics of finite sets , 1989, North-Holland mathematical library.

[93]  Tom Gilb The pre-natal death of the CIS project: A software disaster story , 1988, J. Syst. Softw..

[94]  Dov M. Gabbay,et al.  The Declarative Past and Imperative Future: Executable Temporal Logic for Interactive Systems , 1987, Temporal Logic in Specification.

[95]  Manfred Broy,et al.  Das neue V-Modell® XT , 2005, Informatik-Spektrum.

[96]  Dean Allemang,et al.  The Computational Complexity of Abduction , 1991, Artif. Intell..

[97]  Klaus Schneider,et al.  From PSL to LTL: A Formal Validation in HOL , 2005, TPHOLs.

[98]  Vladimir Lifschitz,et al.  The logic of common sense , 1995, CSUR.

[99]  C. A. R. Hoare,et al.  An axiomatic basis for computer programming , 1969, CACM.

[100]  Nicolas Markey,et al.  Temporal logic with past is exponentially more succinct, Concurrency Column , 2003, Bull. EATCS.

[101]  Fabio Somenzi,et al.  An Algorithm for Strongly Connected Component Analysis in n log n Symbolic Steps , 2000, Formal Methods Syst. Des..

[102]  Koushik Sen,et al.  Rule-Based Runtime Verification , 2004, VMCAI.

[103]  Simin Nadjm-Tehrani,et al.  Verification of Embedded Systems Using Synchronous Observers , 1996, FTRTFT.

[104]  Avner Landver,et al.  The ForSpec Temporal Logic: A New Temporal Property-Specification Language , 2002, TACAS.

[105]  David L. Dill,et al.  Timing Assumptions and Verification of Finite-State Concurrent Systems , 1989, Automatic Verification Methods for Finite State Systems.

[106]  John C. Knight Software Challenges in Aviation Systems , 2002, SAFECOMP.

[107]  Martin Leucker,et al.  Monitoring of Real-Time Properties , 2006, FSTTCS.

[108]  Brian C. Williams,et al.  Diagnosing Complex Systems with Software-Extended Behavior using Constraint Optimization , 2005 .

[109]  M. Oliver,et al.  Structure and Hierarchy in Real-Time Systems , 2002 .

[110]  Joseph Sifakis,et al.  Specification and verification of concurrent systems in CESAR , 1982, Symposium on Programming.

[111]  Thomas A. Henzinger,et al.  The benefits of relaxing punctuality , 1991, PODC '91.

[112]  Michael Fisher,et al.  A Resolution Method for Temporal Logic , 1991, IJCAI.

[113]  Klaus Havelund,et al.  Verify Your Runs , 2005, VSTTE.

[114]  Thomas A. Henzinger,et al.  Event-Clock Automata: A Determinizable Class of Timed Automata , 1999, Theor. Comput. Sci..

[115]  Perdita Stevens,et al.  Observational Mu-Calculus , 1999 .

[116]  Philip Wadler,et al.  The essence of functional programming , 1992, POPL '92.

[117]  Emile Ettedgui,et al.  Safety in the Skies: Personnel and Parties in NTSB Aviation Accident Investigations-Master Volume , 1999 .

[118]  Pierre Wolper The meaning of “formal” , 1996, CSUR.

[119]  Bjarne Stroustrup,et al.  C++ Programming Language , 1986, IEEE Softw..

[120]  Meir M. Lehman,et al.  Program evolution and its impact on software engineering , 1976, ICSE '76.

[121]  Doron Drusinsky,et al.  The Temporal Rover and the ATG Rover , 2000, SPIN.

[122]  G. Plotkin,et al.  Proof, language, and interaction: essays in honour of Robin Milner , 2000 .

[123]  Harald Heinecke,et al.  AUTomotive Open System ARchitecture - An Industry-Wide Initiative to Manage the Complexity of Emerging Automotive E/E-Architectures , 2004 .

[124]  Rolf Isermann,et al.  Model-based fault-detection and diagnosis - status and applications , 2004, Annu. Rev. Control..

[125]  W E Vesely,et al.  Fault Tree Handbook , 1987 .

[126]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[127]  Leszek A. Maciaszek Requirements analysis and system design: developing information systems with UML , 2001 .

[128]  Pierre-Yves Schobbens,et al.  The Logic of Event Clocks - Decidability, Complexity and Expressiveness , 1998, J. Autom. Lang. Comb..

[129]  Gianfranco Lamperti,et al.  Diagnosis of Large Active Systems , 1999, Artif. Intell..

[130]  Raymond Reiter,et al.  A Logic for Default Reasoning , 1987, Artif. Intell..

[131]  Charles E Ebeling,et al.  An Introduction to Reliability and Maintainability Engineering , 1996 .

[132]  Dexter Kozen,et al.  On Kleene Algebras and Closed Semirings , 1990, MFCS.

[133]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[134]  Bernhard Schätz,et al.  AutoFocus: A Tool for Distributed Systems Specification , 1996, FTRTFT.

[135]  Stavros Tripakis,et al.  Analysis of Timed Systems Using Time-Abstracting Bisimulations , 2001, Formal Methods Syst. Des..

[136]  Stephan Merz,et al.  Model Checking , 2000 .

[137]  Joël Ouaknine,et al.  On the decidability of metric temporal logic , 2005, 20th Annual IEEE Symposium on Logic in Computer Science (LICS' 05).

[138]  Algirdas Avizienis,et al.  Design of fault-tolerant computers , 1967, AFIPS '67 (Fall).

[139]  Perdita Stevens,et al.  Enriching OCL Using Observational Mu-Calculus , 2002, FASE.

[140]  Orna Kupferman,et al.  Model Checking of Safety Properties , 1999, Formal Methods Syst. Des..

[141]  Bernhard Schätz,et al.  Integrierte Entwicklung von Automotive-Software mit AutoFocus , 2005, Informatik - Forschung und Entwicklung.

[142]  Kenneth L. McMillan,et al.  Symbolic model checking: an approach to the state explosion problem , 1992 .

[143]  J. R. Büchi On a Decision Method in Restricted Second Order Arithmetic , 1990 .

[144]  Melvin Fitting,et al.  First-Order Logic and Automated Theorem Proving , 1990, Graduate Texts in Computer Science.

[145]  Dana Fisman,et al.  The Temporal Logic Sugar , 2001, CAV.

[146]  John D. Sterman,et al.  All models are wrong: reflections on becoming a systems scientist† , 2002 .

[147]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools , 1986, Addison-Wesley series in computer science / World student series edition.

[148]  Sharad Malik,et al.  Chaff: engineering an efficient SAT solver , 2001, Proceedings of the 38th Design Automation Conference (IEEE Cat. No.01CH37232).

[149]  John Rushby,et al.  Formal Methods and the Certification of Critical Systems , 2004 .

[150]  David Poole,et al.  Normality and Faults in Logic-Based Diagnosis , 1989, IJCAI.

[151]  Pierre Wolper,et al.  An Automata-Theoretic Approach to Automatic Program Verification (Preliminary Report) , 1986, LICS.

[152]  Anneke Kleppe,et al.  The object constraint language: precise modeling with UML , 1998 .

[153]  E. Emerson,et al.  Modalities for model checking (extended abstract): branching time strikes back , 1985, ACM-SIGACT Symposium on Principles of Programming Languages.

[154]  M. G. Jansen,et al.  Automatic Reuse of Knowledge: A Theory , 1999 .

[155]  Amir Pnueli,et al.  Checking that finite state concurrent programs satisfy their linear specification , 1985, POPL.

[156]  Fred B. Schneider Decomposing Properties into Safety and Liveness Using Predicate Logic. , 1987 .

[157]  Donald W. Loveland,et al.  A machine program for theorem-proving , 2011, CACM.

[158]  Klaus Havelund,et al.  Using Runtime Analysis to Guide Model Checking of Java Programs , 2013, SPIN.

[159]  James C. Corbett,et al.  Expressing checkable properties of dynamic systems: the Bandera Specification Language , 2002, International Journal on Software Tools for Technology Transfer.

[160]  Marcel Staroswiecki,et al.  AI and Automatic Control Approaches of Model-Based Diagnosis: Links and Underlying Hypotheses , 2000 .

[161]  Ofer Strichman,et al.  Bounded model checking , 2003, Adv. Comput..

[162]  David Poole,et al.  Default logic , 1994 .

[163]  Olivier Finkel,et al.  Borel hierarchy and omega context free languages , 2003, Theor. Comput. Sci..

[164]  Manfred Broy,et al.  Mathematical System Models as a Basis of Software Engineering , 1995, Computer Science Today.

[165]  Raymond Reiter On Closed World Data Bases , 1977, Logic and Data Bases.

[166]  Donald E. Knuth,et al.  The art of computer programming, volume 3: (2nd ed.) sorting and searching , 1998 .

[167]  Thomas A. Henzinger,et al.  A really temporal logic , 1989, 30th Annual Symposium on Foundations of Computer Science.

[168]  Johan de Kleer,et al.  Problem Solving with the ATMS , 1986, Artif. Intell..

[169]  Bernhard Schätz,et al.  Model-Based Requirements Engineering with AutoRAID , 2005, GI Jahrestagung.

[170]  Leslie Lamport,et al.  Proving the Correctness of Multiprocess Programs , 1977, IEEE Transactions on Software Engineering.

[171]  Sriram K. Rajamani,et al.  The SLAM project: debugging system software via static analysis , 2002, POPL '02.

[172]  S C Kleene,et al.  Representation of Events in Nerve Nets and Finite Automata , 1951 .

[173]  Raymond Reiter,et al.  Foundations of Assumption-based Truth Maintenance Systems: Preliminary Report , 1987, AAAI.

[174]  M. Fitting First-order logic and automated theorem proving (2nd ed.) , 1996 .

[175]  Dimitra Giannakopoulou,et al.  Runtime Analysis of Linear Temporal Logic Specifications , 2001 .

[176]  Klaus Havelund,et al.  Java PathFinder, A Translator from Java to Promela , 1999, SPIN.

[177]  Gerard J. Holzmann,et al.  Design and validation of computer protocols , 1991 .

[178]  Martin Leucker,et al.  Runtime verification revisited , 2005 .

[179]  Andreas Bauer Simplifying Diagnosis Using LSAT: A Propositional Approach to Reasoning from First Principles , 2005, CPAIOR.

[180]  Rolf Isermann Model-based fault-detection and diagnosis - status and applications § , 2004 .

[181]  Grigore Rosu,et al.  Monitoring Java Programs with Java PathExplorer , 2001, RV@CAV.

[182]  Edmund M. Clarke,et al.  Formal Methods: State of the Art and Future Directions Working Group Members , 1996 .

[183]  Peter Struss,et al.  Model-Based Systems in the Automotive Industry , 2004, AI Mag..

[184]  Koushik Sen,et al.  Efficient decentralized monitoring of safety in distributed systems , 2004, Proceedings. 26th International Conference on Software Engineering.

[185]  Donald Ervin Knuth,et al.  The Art of Computer Programming , 1968 .

[186]  Joël Ouaknine,et al.  Completeness and Complexity of Bounded Model Checking , 2004, VMCAI.

[187]  Johan Anthory Willem Kamp,et al.  Tense logic and the theory of linear order , 1968 .

[188]  Pierre Wolper The meaning of “formal”: from weak to strong formal methods , 1997, International Journal on Software Tools for Technology Transfer.

[189]  H. Schneider Failure mode and effect analysis : FMEA from theory to execution , 1996 .

[190]  A. Prasad Sistla,et al.  Safety, liveness and fairness in temporal logic , 1994, Formal Aspects of Computing.

[191]  Jonathan Streit,et al.  SALT - Structured Assertion Language for Temporal Logic , 2006, ICFEM.

[192]  Virginie Wiels,et al.  Using Formal Verification Techniques to Reduce Simulation and Test Effort , 2001, FME.

[193]  David Gries,et al.  A Note on a Standard Strategy for Developing Loop Invariants and Loops , 1982, Sci. Comput. Program..

[194]  Robert McNaughton,et al.  Testing and Generating Infinite Sequences by a Finite Automaton , 1966, Inf. Control..

[195]  Edsger W. Dijkstra,et al.  Structured programming , 1972, A.P.I.C. Studies in data processing.

[196]  Fahiem Bacchus,et al.  Using SAT in QBF , 2005, CP.

[197]  Manfred Broy,et al.  AutoFocus– Ein Werkzeugprototyp zur Entwicklung eingebetteter Systeme , 1999, Informatik Forschung und Entwicklung.

[198]  Howard E. McCurdy Faster, Better, Cheaper , 2001 .

[199]  Peyton Jones,et al.  Haskell 98 language and libraries : the revised report , 2003 .

[200]  Barry W. Boehm,et al.  Understanding and Controlling Software Costs , 1988, IEEE Trans. Software Eng..

[201]  F. Gärtner Formale Grundlagen der Fehlertoleranz in verteilten Systemen , 2001 .

[202]  Pierre-Yves Schobbens,et al.  State Clock Logic: A Decidable Real-Time Logic , 1997, HART.

[203]  Nancy G. Leveson,et al.  Role of Software in Spacecraft Accidents , 2004 .

[204]  Alexander Pretschner Zum modellbasierten funktionalen Test reaktiver Systeme , 2003 .

[205]  Maria Grazia Scutellà,et al.  Polynomially Solvable Satisfiability Problems , 1988, Inf. Process. Lett..

[206]  Andreas Bauer,et al.  Tool-support for the analysis of hybrid systems and models , 2007, 2007 Design, Automation & Test in Europe Conference & Exhibition.

[207]  Hoang Pham,et al.  Handbook of reliability engineering , 2013 .

[208]  Albert Benveniste Non-massive, Non-high Performance, Distributed Computing: Selected Issues , 2002, Euro-Par.

[209]  William F. Clocksin,et al.  Programming in Prolog , 1981, Springer Berlin Heidelberg.

[210]  Yannick Pencolé,et al.  Diagnosis of discrete-event systems using binary decision diagrams , 2004 .

[211]  Wolfgang Weck,et al.  The Greybox Approach: When Blackbox Specifications Hide Too Much , 1999 .

[212]  A. Willsky,et al.  Analytical redundancy and the design of robust failure detection systems , 1984 .

[213]  Vladimir Gurvich,et al.  A New Algorithm for the Hypergraph Transversal Problem , 2005, COCOON.

[214]  John McCarthy,et al.  Applications of Circumscription to Formalizing Common Sense Knowledge , 1987, NMR.

[215]  Grigore Rosu,et al.  Monitoring programs using rewriting , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[216]  Nicolas Halbwachs,et al.  Synchronous Observers and the Verification of Reactive Systems , 1993, AMAST.

[217]  Christos G. Cassandras,et al.  Discrete event systems : modeling and performance analysis , 1993 .

[218]  J. Harrison Introduction to Logic and Automated Theorem Proving , 2007 .