Protocol testing: review of methods and relevance for software testing

Communication protocols are the rules that govern the communication between the different components within a distributed computer system. Since protocols are implemented in software and/or hardware, the question arises whether the existing hardware and software testing methods would be adequate for the testing of communication protocols. The purpose of this paper is to explain in which way the problem of testing protocol implementations is different from the usual problem of software testing. We review the major results in the area of protocol testing and discuss in which way these methods may also be relevant in the more general context of software testing.

[1]  E BryantRandal.  Graph-Based Algorithms for Boolean Function Manipulation , 1986 .

[2]  Son T. Vuong,et al.  On Tools Supporting the Use of Formal Description Techniques in Protocol Development , 1993, Comput. Networks ISDN Syst..

[3]  M. P. Vasilevskii.  Failure diagnosis of automata , 1973 .

[4]  Ming T. Liu,et al.  Generating test cases for EFSM with given fault models , 1993, IEEE INFOCOM '93 The Conference on Computer Communications, Proceedings.

[5]  Rob J. van Glabbeek,et al.  The Linear Time - Branching Time Spectrum II , 1993, CONCUR.

[6]  Ming T. Liu.  Protocol Engineering , 1989, Adv. Comput..

[7]  Jan Tretmans,et al.  A Framework for Test Selection , 1991, PSTV.

[8]  Robert L. Probert,et al.  A Comprehensive Software Environment for Developing Standardized Conformance Test Suites , 1989, Comput. Networks ISDN Syst..

[9]  Paul D. Amer,et al.  Test Case Generation for Protocols Specified in Estelle , 1990, FORTE.

[10]  Alexandre Petrenko,et al.  Test Selection Based on Communicating Nondeterministic Finite-State Machines Using a Generalized WP-Method , 1994, IEEE Trans. Software Eng..

[11]  F. C. Hennine.  Fault detecting experiments for sequential circuits , 1964, SWCT 1964.

[12]  Son T. Vuong,et al.  An Overview of ASN.1 , 1992, Comput. Networks ISDN Syst..

[13]  Rom Langerak.  A Testing Theory for LOTOS using Deadlock Detection , 1989, PSTV.

[14]  N. V. Evtushenko,et al.  Method of constructing a test experiment for an arbitrary deterministic automation , 1991 .

[15]  A. Church.  Edward F. Moore. Gedanken-experiments on sequential machines. Automata studies , edited by C. E. Shannon and J. McCarthy, Annals of Mathematics studies no. 34, litho-printed, Princeton University Press, Princeton1956, pp. 129–153. , 1958, Journal of Symbolic Logic.

[16]  Nina Yevtushenko,et al.  Test Suite Generation from a FSM with a Given Type of Implementation Errors , 1992, PSTV.

[17]  Alexandre Petrenko,et al.  Synthesis of test experiments in some classes of automata , 1991 .

[18]  Alexandre Petrenko,et al.  Conformance Testing of Protocol Machines without Reset , 1993, PSTV.

[19]  Gregor von Bochmann.  Protocol Specification for OSI , 1989, Comput. Networks ISDN Syst..

[20]  Mostafa Hashem Sherif,et al.  Protocol modeling for conformance testing: Case study for the ISDN LAPD protocol , 1990, AT&T Technical Journal.

[21]  Hasan Ural,et al.  Formal methods for test sequence generation , 1992, Comput. Commun..

[22]  D. Rayner.  OSI Conformance Testing , 1987, Comput. Networks.

[23]  R. J. vanGlabbeek.  The linear time - branching time spectrum , 1990 .

[24]  G. Bochmann,et al.  Conformance Relations and Test Derivation , 1993, Protocol Test Systems.

[25]  Guy Leduc,et al.  Conformance Relation, Associated Equivalence, and New Canonical Tester in LOTOS , 1991, PSTV.

[26]  Teruo Higashino,et al.  Automatic Analysis and Test Case Derivation for a Restricted Class of LOTOS Expressions with Data Parameters , 1994, IEEE Trans. Software Eng..

[27]  Gregor von Bochmann,et al.  Distributed Observation and FIFO Queues , 1990, FORTE.

[28]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[29]  Daniel Hoffman,et al.  A case study in class testing , 1993, CASCON.

[30]  Edward F. Moore,et al.  Gedanken-Experiments on Sequential Machines , 1956 .

[31]  Robert K. Brayton,et al.  The maximum set of permissible behaviors for FSM networks , 1993, ICCAD.

[32]  Khalil Drira,et al.  Testability of a Communicating System Through an Environment , 1993, TAPSOFT.

[33]  Deepinder P. Sidhu,et al.  Formal Methods for Protocol Testing: A Detailed Study , 1989, IEEE Trans. Software Eng..

[34]  Kwang-Ting Cheng,et al.  Automatic Functional Test Generation Using The Extended Finite State Machine Model , 1993, 30th ACM/IEEE Design Automation Conference.

[35]  Dieter Hogrefe,et al.  The CCITT-Specification and Description Language SDL , 1989, Comput. Networks.

[36]  D. J. Robson,et al.  The Testing of Object-Oriented Programs , 1993 .

[37]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[38]  BolognesiTommaso,et al.  Introduction to the ISO specification language LOTOS , 1987 .

[39]  G. Bochmann,et al.  Fault Models in Testing , 1991, Protocol Test Systems.

[40]  Hasan Ural.  Test sequence selection based on static data flow analysis , 1987, Comput. Commun..

[41]  Richard J. Linn,et al.  Automatic generation of test scenario skeletons from protocol specifications written in ESTELLE , 1986 .

[42]  Alfred V. Aho,et al.  An optimization technique for protocol conformance test generation based on UIO sequences and rural Chinese postman tours , 1991, IEEE Trans. Commun..

[43]  Hasan Ural,et al.  Test Generation by Exposing Control and Data Dependencies Within System Specifications in SDL , 1993, FORTE.

[44]  Gregor von Bochmann,et al.  A Test Design Methodology for Protocol Testing , 1987, IEEE Transactions on Software Engineering.

[45]  Gregor von Bochmann,et al.  Synchronization and Specification Issues in Protocol Testing , 1984, IEEE Trans. Commun..

[46]  Sanjoy Paul,et al.  Generating Conformance Test Sequences for Combined Control and Data Flow of Communication Protocols , 1992, PSTV.

[47]  Krishan K. Sabnani,et al.  An experience in estimating fault coverage of a protocol test , 1988, IEEE INFOCOM '88,Seventh Annual Joint Conference of the IEEE Computer and Communcations Societies. Networks: Evolution or Revolution?.

[48]  Ferhat Khendek,et al.  Test Selection Based on Finite State Models , 1991, IEEE Trans. Software Eng..

[49]  Gregor von Bochmann,et al.  Generating Synchronizable Test Sequences Based on Finite State Machine with Distributed Ports , 1993, Protocol Test Systems.

[50]  Nina Yevtushenko,et al.  Nondeterministic State Machines in Protocol Conformance Testing , 1993, Protocol Test Systems.

[51]  Deepinder P. Sidhu,et al.  Fault Coverage of UIO-based Methods for Protocol Testing , 1993, Protocol Test Systems.

[52]  Alexandre Petrenko,et al.  Fault coverage analysis in respect to an FSM specification , 1994, Proceedings of INFOCOM '94 Conference on Computer Communications.

[53]  Robert K. Brayton,et al.  The maximum set of permissible behaviors for FSM networks , 1993, Proceedings of 1993 International Conference on Computer Aided Design (ICCAD).

[54]  Tommaso Bolognesi,et al.  Tableau methods to describe strong bisimilarity on LOTOS processes involving pure interleaving and enabling , 1994, FORTE.

[55]  David Lee,et al.  Testing finite state machines , 1991, STOC '91.

[56]  Gregor von Bochmann,et al.  Testing Non-Deterministic State Machines with Fault Coverage , 1991, Protocol Test Systems.

[57]  Jan Tretmans,et al.  A Formal Approach to Conformance Testing , 1993, Protocol Test Systems.

[58]  Gregor von Bochmann,et al.  TESTL: An Environment for Incremental Test Suite Design Based on Finite-State Models , 1991, Protocol Test Systems.

[59]  Son T. Vuong,et al.  A Framework for the Design for Testability of Communication Protocols , 1993, Protocol Test Systems.

[60]  Behcet Sarikaya,et al.  Standard conformance test specification language TTCN , 1992 .

[61]  Jan Tretmans,et al.  Protocol Conformance Testing: A Formal Perspective on ISO IS-9646 , 1991, Protocol Test Systems.

[62]  Fabio Somenzi,et al.  Fast sequential ATPG based on implicit state enumeration , 1991, 1991, Proceedings. International Test Conference.

[63]  Clazien D. Wezeman.  The CO-OP Method for Compositional Derivation of Conformance Testers , 1989, PSTV.

[64]  Hasan Ural,et al.  The Synchronization Problem in Protocol Testing and its Complexity , 1991, Inf. Process. Lett..

[65]  Alexandre Petrenko.  Checking Experiments with Protocol Machines , 1991, Protocol Test Systems.

[66]  Robin Milner,et al.  A Calculus of Communicating Systems , 1980, Lecture Notes in Computer Science.

[67]  Stanislaw Budkowski,et al.  An Introduction to Estelle: A Specification Language for Distributed Systems , 1987, Comput. Networks.

[68]  Rudie Alderden,et al.  COOPER - The Compositional Construction of a Canonical Tester , 1989, Formal Techniques for (Networked and) Distributed Systems.

[69]  C. A. R. Hoare,et al.  Communicating sequential processes , 1978, CACM.