Working together : Formal Methods and Testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding the ways in which formal methods and testing complement one another.

[1]  Anders P. Ravn,et al.  Hybrid Control of a Robot - A Case Study , 1994, Hybrid Systems.

[2]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[3]  Monique Snoeck,et al.  Using design patterns to reveal the competence of object-oriented methods in system-level design , 1999 .

[4]  David W. Binkley,et al.  Program slicing , 2008, 2008 Frontiers of Software Maintenance.

[5]  Simon Burton.  Towards Automated Unit Testing of Statechart Implementations , 1999 .

[6]  Bruno Legeard,et al.  Automated Boundary Testing from Z and B , 2002, FME.

[7]  Johann Schumann,et al.  Generating statechart designs from scenarios , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[8]  Simin Nadjm-Tehrani,et al.  Proving dynamic properties in an aerospace application , 1995, Proceedings 16th IEEE Real-Time Systems Symposium.

[9]  George S. Avrunin,et al.  Symbolic Model Checking Using Algebraic Geometry , 1996, CAV.

[10]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[11]  Jonathan P. Bowen,et al.  Reinforced Condition/Decision Coverage (RC/DC): A New Criterion for Software Testing , 2002, ZB.

[12]  J. Michael Spivey,et al.  Understanding Z : A specification language and its formal semantics , 1985, Cambridge tracts in theoretical computer science.

[13]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[14]  Michael D. Ernst,et al.  Dynamically discovering likely program invariants , 2000 .

[15]  Mirko Conrad,et al.  Test case design based on Z and the classification-tree method , 1997, First IEEE International Conference on Formal Engineering Methods.

[16]  Colin Stirling,et al.  Modal and temporal logics , 1993, LICS 1993.

[17]  Anthony J. H. Simons,et al.  Use cases considered harmful , 1999, Proceedings Technology of Object-Oriented Languages and Systems. TOOLS 29 (Cat. No.PR00275).

[18]  Alexandre Petrenko,et al.  Test Selection Based on Communicating Nondeterministic Finite-State Machines Using a Generalized WP-Method , 1994, IEEE Trans. Software Eng..

[19]  William E. Lorensen,et al.  Object-Oriented Modeling and Design , 1991, TOOLS.

[20]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[21]  Philip Alan Stocks,et al.  Applying formal methods to software testing , 1993 .

[22]  Anthony Hall,et al.  Correctness by Construction: Developing a Commercial Secure System , 2002, IEEE Softw..

[23]  Panos J. Antsaklis,et al.  Hybrid Systems II , 1994, Lecture Notes in Computer Science.

[24]  Lex Heerink,et al.  Factorized Test Generation for Multi-Input/Output Transition Systems , 1998, IWTCS.

[25]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[26]  Iso. Lotos,et al.  A Formal Description Technique Based on the Temporal Ordering of Observational Behaviour , 1985 .

[27]  Edsger W. Dijkstra,et al.  Notes on structured programming , 1970 .

[28]  David A. Schmidt,et al.  Denotationaisemantics: a methodology for language development , 1986 .

[29]  Anthony J. H. Simons.  On the Compositional Properties of UML Statechart Diagrams , 2000, Rigorous Object-Oriented Methods.

[30]  Guney Gonenc,et al.  A Method for the Design of Fault Detection Experiments , 1970, IEEE Transactions on Computers.

[31]  David L. Dill,et al.  The Murphi Verification System , 1996, CAV.

[32]  Dieter Hogrefe,et al.  Autolink - Putting SDL-Based Test Generation Into Practice , 1998, IWTCS.

[33]  Mark Harman,et al.  Slicing, I/O and the Implicit State , 1997, AADEBUG.

[34]  Zohar Manna,et al.  Temporal Verification of Reactive Systems , 1995, Springer New York.

[35]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[36]  Michael Benedikt,et al.  Model Checking of Unrestricted Hierarchical State Machines , 2001, ICALP.

[37]  Derek Coleman,et al.  Formal Specification and Design Time Testing , 1990, IEEE Trans. Software Eng..

[38]  Steve King,et al.  The Value of Verification: Positive Experience of Industrial Proof , 1999, World Congress on Formal Methods.

[39]  Simon Burton.  Automated Testing From Z Specifications , 2000 .

[40]  Amir Pnueli,et al.  Checking that finite state concurrent programs satisfy their linear specification , 1985, POPL.

[41]  Gerard J. Holzmann,et al.  On-the-fly model checking , 1996, CSUR.

[42]  Tommaso Bolognesi,et al.  Tableau methods to describe strong bisimilarity on LOTOS processes involving pure interleaving and enabling , 1994, FORTE.

[43]  James C. Corbett,et al.  Timing Analysis of Ada Tasking Programs , 1996, IEEE Trans. Software Eng..

[44]  Robert M. Hierons.  Checking states and transitions of a set of communicating finite state machines , 2001, Microprocess. Microsystems.

[45]  Patrice Godefroid,et al.  Partial-Order Methods for the Verification of Concurrent Systems , 1996, Lecture Notes in Computer Science.

[46]  Saburo Muroga,et al.  Binary Decision Diagrams , 2000, The VLSI Handbook.

[47]  Florentin Ipate,et al.  Generating Test Sets from Non-Deterministic Stream X-Machines , 2000, Formal Aspects of Computing.

[48]  Paul Ammann,et al.  Test Generation and Recognition with Formal Methods , 2000 .

[49]  Alfred V. Aho,et al.  An optimization technique for protocol conformance test generation based on UIO sequences and rural Chinese postman tours , 1991, IEEE Trans. Commun..

[50]  David W. Binkley,et al.  The application of program slicing to regression testing , 1998, Inf. Softw. Technol..

[51]  Mark Harman,et al.  Using program slicing to simplify testing , 1995, Softw. Test. Verification Reliab..

[52]  Robert M. Hierons,et al.  Testing from a Finite-State Machine: Extending Invertibility to Sequences , 1997, Comput. J..

[53]  Edsger W. Dijkstra,et al.  A Discipline of Programming , 1976 .

[54]  Richard G. Hamlet,et al.  Data Abstraction, Implementation, Specification, and Testing , 1981, TOPL.

[55]  Angelo Gargantini,et al.  Using model checking to generate tests from requirements specifications , 1999, ESEC/FSE-7.

[56]  P. David Stotts,et al.  Daistish: systematic algebraic testing for OO programs in the presence of side-effects , 1996, ISSTA '96.

[57]  Dexter Kozen,et al.  Results on the Propositional µ-Calculus , 1982, ICALP.

[58]  John D. McGregor,et al.  Integrated object-oriented testing and development processes , 1994, CACM.

[59]  William G. Griswold,et al.  Quickly detecting relevant program invariants , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[60]  Bernhard K. Aichernig.  Test-Case Calculation through Abstraction , 2001, FME.

[61]  Girish Bhat,et al.  Efficient on-the-fly model checking for CTL , 1995, Proceedings of Tenth Annual IEEE Symposium on Logic in Computer Science.

[62]  Hilary Putnam,et al.  A Computing Procedure for Quantification Theory , 1960, JACM.

[63]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[64]  Hans-Martin Hörcher,et al.  Improving Software Tests Using Z Specifications , 1995, ZUM.

[65]  J. van Leeuwen.  ZUM ’98: The Z Formal Specification Notation , 1998, Lecture Notes in Computer Science.

[66]  Ian MacColl,et al.  Testing Matis: a Case Study on Speciication-based Testing of Interactive Systems , 1998 .

[67]  Zvi Kohavi,et al.  Checking experiments for sequential machines , 1974, Inf. Sci..

[68]  Zohar Manna,et al.  Verifying Hybrid Systems , 1992, Hybrid Systems.

[69]  S. Seshu,et al.  Introduction to the theory of finite-state machines , 1963 .

[70]  M. R. Woodward,et al.  Errors in algebraic specifications and an experimental mutation testing tool , 1993, Softw. Eng. J..

[71]  Clazien D. Wezeman,et al.  Deriving Tests for Objects Specified in Z , 1992, Z User Workshop.

[72]  Wang Yi,et al.  UPPAAL - a Tool Suite for Automatic Verification of Real-Time Systems , 1996, Hybrid Systems.

[73]  Thomas A. Henzinger,et al.  A Space-Efficient On-the-fly Algorithm for Real-Time Model Checking , 1996, CONCUR.

[74]  Robert M. Hierons,et al.  Adaptive Testing of a Deterministic Implementation Against a Nondeterministic Finite State Machine , 1998, Comput. J..

[75]  Jan Tretmans,et al.  Conformance Testing with Labelled Transition Systems: Implementation Relations and Test Generation , 1996, Comput. Networks ISDN Syst..

[76]  Marie-Claude Gaudel,et al.  Using algebraic specifications in software testing: A case study on the software of an automatic subway , 1993, J. Syst. Softw..

[77]  Marie-Claude Gaudel,et al.  Software testing based on formal specifications: a theory and a tool , 1991, Softw. Eng. J..

[78]  Ian Graham,et al.  30 Things that Go Wrong in Object Modelling with UML 1.3 , 1999, Behavioral Specifications of Businesses and Systems.

[79]  Akash Deshpande,et al.  The SHIFT Programming Language and Run-time System for Dynamic Networks of Hybrid Automata , 1997 .

[80]  Rance Cleaveland,et al.  Tableau-based model checking in the propositional mu-calculus , 1990, Acta Informatica.

[81]  John R. Callahan,et al.  Automated Software Testing Using Model-Checking , 1996 .

[82]  Jonathan P. Bowen,et al.  Formalization of software testing criteria using the Z notation , 2001, 25th Annual International Computer Software and Applications Conference. COMPSAC 2001.

[83]  Rajeev Alur,et al.  Timing Analysis in COSPAN , 1996, Hybrid Systems.

[84]  Tsong Yueh Chen,et al.  TACCLE: a methodology for object-oriented software testing at the class and cluster levels , 2001, TSEM.

[85]  Thierry Jéron,et al.  Automated test generation from SDL specifications , 1999, SDL Forum.

[86]  G. Bochmann,et al.  Testing deterministic implementations from nondeterministic FSM specifications , 1996 .

[87]  Robert W. Floyd,et al.  Assigning Meanings to Programs , 1993 .

[88]  Bruno Marre,et al.  Test sequences generation from LUSTRE descriptions: GATEL , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[89]  Clazien D. Wezeman.  The CO-OP Method for Compositional Derivation of Conformance Testers , 1989, PSTV.

[90]  Laurent Fribourg,et al.  Test sets generation from algebraic specifications using logic programming , 1986, J. Syst. Softw..

[91]  Stelvio Cimato,et al.  Engineering Formal Requirements: Analysis and Testing , 1996, SEKE.

[92]  Bruno Legeard,et al.  A Comparison of the BTT and TTF Test-Generation Methods , 2002, ZB.

[93]  Colin Stirling,et al.  Modal Logics and mu-Calculi: An Introduction , 2001, Handbook of Process Algebra.

[94]  Robert M. Hierons,et al.  Reduced Length Checking Sequences , 2002, IEEE Trans. Computers.

[95]  Thomas A. Henzinger,et al.  HYTECH: the next generation , 1995, Proceedings 16th IEEE Real-Time Systems Symposium.

[96]  Florentin Ipate,et al.  An integration testing method that is proved to find all faults , 1997, Int. J. Comput. Math..

[97]  Anneke Kleppe,et al.  The object constraint language: precise modeling with UML , 1998 .

[98]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[99]  Jonathan P. Bowen,et al.  ZB 2000: Formal Specification and Development in Z and B , 2000, Lecture Notes in Computer Science.

[100]  Bernhard K. Aichernig,et al.  Validating Voice Communication Requirements Using Lightweight Formal Methods , 2000, IEEE Softw..

[101]  F. C. Hennine.  Fault detecting experiments for sequential circuits , 1964, SWCT 1964.

[102]  Julie Candler,et al.  SMART CARS, SMART ROADS , 1996 .

[103]  Kai Koskimies,et al.  Automated Support for Modeling OO Software , 1998, IEEE Softw..

[104]  Shaoying Liu.  Verifying Consistency and Validity of Formal Specifications by Testing , 1999, World Congress on Formal Methods.

[105]  Richard Atterer.  Automatic Test Data Generation From VDM-SL Specifications , 2000 .

[106]  Bernhard K. Aichernig,et al.  Automated Black-Box Testing with Abstract VDM Oracles , 1999, SAFECOMP.

[107]  Huaikou Miao,et al.  An approach to testing the nonexistence of initial state in Z specifications , 1999, Proceedings Eighth Asian Test Symposium (ATS'99).

[108]  Marc Phalippou.  Executable Testers , 1993, Protocol Test Systems.

[109]  David L. Dill,et al.  Java model checking , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[110]  Shaoying Liu,et al.  Generating test data from SOFL specifications , 1999, J. Syst. Softw..

[111]  John A. Clark,et al.  Automated program flaw finding using simulated annealing , 1998, ISSTA '98.

[112]  Charles E. Leiserson,et al.  Detecting data races in Cilk programs that use locks , 1998, SPAA '98.

[113]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[114]  Mark Utting,et al.  Data structures for Z testing tools. , 2001 .

[115]  Harmen-Hinrich Sthamer,et al.  The automatic generation of software test data using genetic algorithms , 1995 .

[116]  Joachim Wegener,et al.  Testing real-time systems using genetic algorithms , 1997, Software Quality Journal.

[117]  G. Winskel,et al.  A Compositional Proof System for the Modal mu-Calculus , 1994 .

[118]  Sérgio Vale Aguiar Campos,et al.  Symbolic Model Checking , 1993, CAV.

[119]  Gerald W. Both,et al.  Object-oriented analysis and design with applications , 1994 .

[120]  Robert L. Grossman,et al.  Timed Automata , 1999, CAV.

[121]  Bernhard K. Aichernig,et al.  Systematic Black-Box Testing of Computer-Based Systems through Formal Abstraction Techniques , 2001 .

[122]  Pankaj Jalote,et al.  Automated testcase generation for data abstraction , 1988, Proceedings COMPSAC 88: The Twelfth Annual International Computer Software & Applications Conference.

[123]  Doron A. Peled,et al.  Ten Years of Partial Order Reduction , 1998, CAV.

[124]  Susan Horwitz,et al.  Incremental program testing using program dependence graphs , 1993, POPL '93.

[125]  Armin Biere,et al.  Symbolic Model Checking without BDDs , 1999, TACAS.

[126]  Paul Ammann,et al.  Using formal methods to derive test frames in category-partition testing , 1994, Proceedings of COMPASS'94 - 1994 IEEE 9th Annual Conference on Computer Assurance.

[127]  Alexandre Petrenko,et al.  Protocol testing: review of methods and relevance for software testing , 1994, ISSTA '94.

[128]  Randal E. Bryant,et al.  Graph-Based Algorithms for Boolean Function Manipulation , 1986, IEEE Transactions on Computers.

[129]  John M. Rushby,et al.  Integrated Formal Verification: Using Model Checking with Automated Abstraction, Invariant Generation, and Theorem Proving , 1999, SPIN.

[130]  Peter Amey.  A language for systems not just software , 2001, SIGAda '01.

[131]  Edmund M. Clarke,et al.  Compositional model checking , 1989, [1989] Proceedings. Fourth Annual Symposium on Logic in Computer Science.

[132]  David A. Carrington,et al.  A Framework for Specification-Based Testing , 1996, IEEE Trans. Software Eng..

[133]  John-Morten Godhavn,et al.  Hybrid Control in Sea Traffic Management Systems , 1995, Hybrid Systems.

[134]  T. Henzinger,et al.  Algorithmic Analysis of Nonlinear Hybrid Systems , 1998, CAV.

[135]  Matthias Grochtmann,et al.  Systematic Testing of Real-Time Systems , 2000 .

[136]  Shaoying Liu,et al.  A GUI and testing tool for SOFL , 2000, Proceedings Seventh Asia-Pacific Software Engeering Conference. APSEC 2000.

[137]  Panos J. Antsaklis,et al.  Hybrid Systems IV , 1997, Lecture Notes in Computer Science.

[138]  Michael Luck,et al.  ZUM '95: The Z Formal Specification Notation, 9th International Conference of Z Users , 1995 .

[139]  Insup Lee,et al.  Automatic Test Generation From Statecharts Using Model Checking , 2001 .

[140]  Kirill Bogdanov.  Automated testing of Harel's statecharts , 2000 .

[141]  Mordechai Ben-Menachem,et al.  Writing effective use cases , 2001, SOEN.

[142]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[143]  Mark Harman,et al.  An overview of program slicing , 2001, Softw. Focus.

[144]  Ivar Jacobson,et al.  The Unified Modeling Language User Guide , 1998, J. Database Manag..

[145]  Thomas A. Henzinger,et al.  HYTECH: a model checker for hybrid systems , 1997, International Journal on Software Tools for Technology Transfer.

[146]  Fabrice Bouquet,et al.  BZ-testing-tools: A tool-set for test generation from Z and B using constraint logic programming , 2002 .

[147]  Jean-Luc Lambert,et al.  A comparative study between linear programming verification (LPV) and other verification methods , 1999, 14th IEEE International Conference on Automated Software Engineering.

[148]  Satoshi Yamane,et al.  The symbolic model-checking for real-time systems , 1996, Proceedings of the Eighth Euromicro Workshop on Real-Time Systems.

[149]  Gerard J. Holzmann,et al.  SOFTWARE TESTING, VERIFICATION AND RELIABILITY , 2022 .

[150]  Edmund M. Clarke,et al.  Formal Methods: State of the Art and Future Directions Working Group Members , 1996 .

[151]  Mark Harman,et al.  Testing Conformance to a Quasi-Non-Deterministic Stream X-Machine , 2000, Formal Aspects of Computing.

[152]  Nina Yevtushenko,et al.  Nondeterministic State Machines in Protocol Conformance Testing , 1993, Protocol Test Systems.

[153]  Rachida Dssouli,et al.  Automatic Test Generation for EFSM-based Systems , 1996 .

[154]  Jeremy Dick,et al.  Making the Most of Formal Specification Through Animation, Testing and Proof , 1997, Sci. Comput. Program..

[155]  Edmund M. Clarke,et al.  Symbolic Model Checking: 10^20 States and Beyond , 1990, Inf. Comput..

[156]  Robert M. Hierons.  Testing from a Z Specification , 1997, Softw. Test. Verification Reliab..

[157]  Pankaj Jalote.  Specification and Testing of Abstract Data Types , 1992, Comput. Lang..

[158]  Alexandre Petrenko,et al.  Fault Model-Driven Test Derivation from Finite State Models: Annotated Bibliography , 2000, MOVEP.

[159]  John A. Clark,et al.  FORTEST: formal methods and testing , 2002, Proceedings 26th Annual International Computer Software and Applications.

[160]  Alexandre Petrenko,et al.  Automating the process of test derivation from SDL specifications , 1997, SDL Forum.

[161]  Edward V. Berard,et al.  Testing Object-Oriented Software , 2021, TOOLS.

[162]  Jean-Raymond Abrial,et al.  The B-book - assigning programs to meanings , 1996 .

[163]  Eerke A. Boiten,et al.  Testing Refinements of State-based Formal Specifications , 1999, Softw. Test. Verification Reliab..

[164]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[165]  Abdullah Mohd Zin,et al.  Probe: a formal specification-based testing system , 1999, ICIS.

[166]  Bryan F. Jones,et al.  Automatic structural testing using genetic algorithms , 1996, Softw. Eng. J..

[167]  Ekkart Rudolph,et al.  Tutorial on Message Sequence Charts , 1996, Comput. Networks ISDN Syst..

[168]  Jexice I. Glasgow,et al.  LUCID: a specification language for distributed systems , 1985, SOEN.

[169]  Krishan K. Sabnani,et al.  A Protocol Test Generation Procedure , 1988, Comput. Networks.

[170]  Ian J. Hayes,et al.  Specification directed module testing , 1986, IEEE Transactions on Software Engineering.

[171]  Paul Ammann,et al.  Using Z specifications in category partition testing , 1992, COMPASS `92 Proceedings of the Seventh Annual Conference on Computer Assurance.

[172]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[173]  Dieter Hogrefe,et al.  The CCITT-Specification and Description Language SDL , 1989, Comput. Networks.

[174]  Souâd Taouil-Traverson,et al.  Preliminary analysis cycle for B-method software development , 1996, Proceedings of EUROMICRO 96. 22nd Euromicro Conference. Beyond 2000: Hardware and Software Design Strategies.

[175]  Stavros Tripakis,et al.  The Tool KRONOS , 1996, Hybrid Systems.

[176]  Erich Mikk,et al.  Compilation of Z Specifications into C for Automatic Test Result Evaluation , 1995, ZUM.

[177]  Thomas Santen,et al.  Automating Test Case Generation from Z Specifications with Isabelle , 1997, ZUM.

[178]  Alan W. Biermann,et al.  Constructing Programs from Example Computations , 1976, IEEE Transactions on Software Engineering.

[179]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[180]  Loe M. G. Feijs,et al.  Test Generation for Intelligent Networks Using Model Checking , 1997, TACAS.

[181]  Leesa Murray,et al.  Formal Derivation of Finite State Machines for Class Testing , 1998, ZUM.

[182]  Marian Gheorghe,et al.  Communicating Stream X-Machines Systems are no more than X-Machines , 1999, J. Univers. Comput. Sci..

[183]  Bernhard K. Aichernig,et al.  Formal specification techniques as a catalyst in validation , 2000, Proceedings. Fifth IEEE International Symposium on High Assurance Systems Engineering (HASE 2000).

[184]  Amnon Naamad,et al.  The STATEMATE semantics of statecharts , 1996, TSEM.

[185]  Alexandre Petrenko,et al.  Synthesis of test experiments in some classes of automata , 1991 .

[186]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[187]  Xiaolin Wu,et al.  On Minimizing the Lengths of Checking Sequences , 1997, IEEE Trans. Computers.

[188]  Ferhat Khendek,et al.  Test Selection Based on Finite State Models , 1991, IEEE Trans. Software Eng..

[189]  Thomas A. Henzinger,et al.  HYTECH: The Cornell HYbrid TECHnology Tool , 1994, Hybrid Systems.

[190]  Richard Gerber,et al.  Symbolic Model Checking of Infinite State Systems Using Presburger Arithmetic , 1997, CAV.

[191]  Christophe Meudec,et al.  Automatic generation of software test cases from formal specifications , 1998 .

[192]  K. Rustan M. Leino,et al.  Extended static checking , 1998, PROCOMET.

[193]  Pretschner,et al.  Testing Concurrent Reactive Systems with Constraint Logic Programming , 2000 .

[194]  Alexandre Petrenko,et al.  Selecting test sequences for partially-specified nondeterministic finite state machines , 1995 .

[195]  John D. Gannon,et al.  Combining Testing with Formal Specifications: A Case Study , 1983, IEEE Transactions on Software Engineering.

[196]  Nicolas Halbwachs,et al.  Verification of Linear Hybrid Systems by Means of Convex Approximations , 1994, SAS.

[197]  Bo Yang,et al.  Protocol conformance test generation using multiple UIO sequences with overlapping , 1990, SIGCOMM '90.

[198]  J. McDermid,et al.  Testing , Proof and Automation . An Integrated Approach , 2000 .

[199]  Robert M. Hierons,et al.  Testing a system specified using Statecharts and Z , 2001, Inf. Softw. Technol..

[200]  Pierre Wolper,et al.  An Automata-Theoretic Approach to Automatic Program Verification (Preliminary Report) , 1986, LICS.

[201]  P.A.V. Hall,et al.  Towards testing with respect to formal specification , 1988 .

[202]  Clemens A. Szyperski,et al.  Component software - beyond object-oriented programming , 2002 .

[203]  James C. Corbett,et al.  Bandera: extracting finite-state models from Java source code , 2000, ICSE.

[204]  Stephan Merz,et al.  Model Checking , 2000 .

[205]  M. R. Woodward,et al.  Assessing the quality of specification-based testing , 1996 .

[206]  E. Allen Emerson,et al.  Temporal and Modal Logic , 1991, Handbook of Theoretical Computer Science, Volume B: Formal Models and Sematics.

[207]  Ken Sakamura,et al.  A specification-based adaptive test case generation strategy for open operating system standards , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[208]  Amir Pnueli,et al.  Translation Validation , 1998, TACAS.

[209]  A. M. Turing,et al.  Checking a large routine , 1989 .

[210]  Jean-Marc Nerson,et al.  Object-Oriented Analysis and Design , 1992, TOOLS.

[211]  Guy Leduc,et al.  Conformance Relation, Associated Equivalence, and New Canonical Tester in LOTOS , 1991, PSTV.

[212]  Phyllis G. Frankl,et al.  The ASTOOT approach to testing object-oriented programs , 1994, TSEM.

[213]  A. W. Heerink,et al.  Developments in Testing Transition Systems , 1997 .

[214]  C. A. R. Hoare,et al.  How Did Software Get So Reliable Without Proof? , 1996, FME.

[215]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[216]  Fausto Giunchiglia,et al.  NUSMV: A New Symbolic Model Verifier , 1999, CAV.

[217]  Daniel Le Métayer,et al.  CASTING: a formally based software test generation method , 1997, First IEEE International Conference on Formal Engineering Methods.

[218]  Robert M. Hierons,et al.  Extending Test Sequence Overlap by Invertibility , 1996, Comput. J..

[219]  Christophe Meudec.  ATGen: automatic test data generation using constraint logic programming and symbolic execution † , 2001, Softw. Test. Verification Reliab..

[220]  Orna Kupferman,et al.  On the Complexity of Branching Modular Model Checking (Extended Abstract) , 1995, CONCUR.

[221]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[222]  James H. Fetzer.  Program verification: the very idea , 1988, CACM.

[223]  Amir Pnueli,et al.  Validation of Optimizing Compilers , 2001 .

[224]  Mark Blackburn,et al.  Automatic generation of test vectors for SCR-style specifications , 1997, Proceedings of COMPASS '97: 12th Annual Conference on Computer Assurance.

[225]  Daniel Jackson,et al.  Finding bugs with a constraint solver , 2000, ISSTA '00.

[226]  Pretschner,et al.  AutoFocus on Constraint Logic Programming , 2000 .

[227]  Thomas A. Henzinger,et al.  Linear Phase-Portrait Approximations for Nonlinear Hybrid Systems , 1996, Hybrid Systems.

[228]  Leesa Murray,et al.  From Object-Z Specifications to ClassBench Test Suites , 2000, Softw. Test. Verification Reliab..

[229]  Bryan F. Jones,et al.  A Strategy for Using Genetic Algorithms to Automate Branch and Fault-Based Testing , 1998, Comput. J..

[230]  Alexandre Petrenko,et al.  Conformance Testing of Protocol Machines without Reset , 1993, PSTV.

[231]  Steve A. Schneider,et al.  Test Case Preparation Using a Prototype , 1998, B.

[232]  Martin Peschke,et al.  Design and Validation of Computer Protocols , 2003 .

[233]  Deepinder P. Sidhu,et al.  Formal Methods for Protocol Testing: A Detailed Study , 1989, IEEE Trans. Software Eng..

[234]  Thomas A. Henzinger,et al.  Using HyTech to Synthesize Control Parameters for a Steam Boiler , 1995, Formal Methods for Industrial Applications.

[235]  Thomas Stauner,et al.  Using HYTECH to Verify an Automative Control System , 1997, HART.

[236]  Florentin Ipate,et al.  Correct systems - building a business process solution , 1998, Applied computing.

[237]  Rance Cleaveland,et al.  A linear-time model-checking algorithm for the alternation-free modal mu-calculus , 1993, Formal Methods Syst. Des..

[238]  Jonathan P. Bowen,et al.  ZB 2002:Formal Specification and Development in Z and B , 2002, Lecture Notes in Computer Science.

[239]  Robin Milner,et al.  Communication and concurrency , 1989, PHI Series in computer science.

[240]  Steve King,et al.  Is Proof More Cost-Effective Than Testing? , 2000, IEEE Trans. Software Eng..

[241]  Orna Grumberg,et al.  Branching-Time Temporal Logic and Tree Automata , 1996, Inf. Comput..

[242]  Susan Stepney.  Testing as Abstraction , 1995, ZUM.

[243]  Alexander Pretschner,et al.  Specification based test sequence generation with propositional logic , 2000, Softw. Test. Verification Reliab..

[244]  David A. Carrington,et al.  A Tale of Two Paradigms: Formal Methods and Software Testing , 1994, Z User Workshop.

[245]  A. S. M. Sajeev,et al.  A Framework for Testing Object-Oriented Software Using Formal Specifications , 1996, Ada-Europe.

[246]  Hélène Waeselynck,et al.  Test Criteria Definition for B Models , 1999, World Congress on Formal Methods.

[247]  C. A. R. Hoare,et al.  Communicating sequential processes , 1978, CACM.

[248]  Klaus Havelund,et al.  Model checking JAVA programs using JAVA PathFinder , 2000, International Journal on Software Tools for Technology Transfer.

[249]  Joseph Sifakis,et al.  Specification and verification of concurrent systems in CESAR , 1982, Symposium on Programming.

[250]  Michael D. Rice,et al.  An approach to architectural analysis and testing , 1998, ISAW '98.

[251]  David Lee,et al.  Testing Finite-State Machines: State Identification and Verification , 1994, IEEE Trans. Computers.

[252]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[253]  Ivar Jacobson,et al.  Object-oriented software engineering - a use case driven approach , 1993, TOOLS.

[254]  Joachim Wegener,et al.  Testing the temporal behavior of real-time software modules using extended evolutionary algorithms , 1999 .

[255]  Brent Hailpern.  Verifying Concurrent Processes Using Temporal Logic , 1982, Lecture Notes in Computer Science.

[256]  Hélène Waeselynck,et al.  B model animation for external verification , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[257]  Jonathan P. Bowen,et al.  ZUM '95: The Z Formal Specification Notation , 1995, Lecture Notes in Computer Science.

[258]  Mike Holcombe,et al.  Testing Strategies for Software Requirements and Design , 1994 .