A survey of communication protocol testing

Academic research has made significant advances in the generation of test sequences from formal specifications, and in the development of computer-aided test tools, with the aim of improving the effectiveness of protocol testing. However, this state-of-the-art research is not necessarily state-of-the-practice; these methods and tools are seldom used in the computer and communications industries. There is a big gap between testing practice and research results published in journals and reported at conference. To help the industrialization of academic techniques, empirical studies and a new orientation towards real issues need to be given higher priority by researchers, and real-life test coverage and field experience need to be more frequently reported [66]. This paper presents a literature survey of communication protocol testing. We have focused our survey on the following five areas: test sequence generation methods, test coverage, fault model and prediction, test tools, and experience reports.

[1]  Tadanori Mizuno,et al.  Strategic Testing Environment with Formal Description Techniques , 1991, IEEE Trans. Computers.

[2]  Son T. Vuong,et al.  On Test Coverage Metrics for Communication Protocols , 1991, Protocol Test Systems.

[3]  Son T. Vuong,et al.  Automated Test Case Selection Based on Test Coverage Metrics , 1992, Protocol Test Systems.

[4]  Siddhartha R. Dalal,et al.  The Automatic Efficient Test Generator (AETG) system , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[5]  Jai-Yong Lee,et al.  A Well-Defined Estelle Specification for the Automatic Test Generation , 1991, IEEE Trans. Computers.

[6]  Deepinder P. Sidhu,et al.  Fault Coverage of UIO-based Methods for Protocol Testing , 1993, Protocol Test Systems.

[7]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[8]  Ian Davidson,et al.  Protocol test systems III : proceedings of the IFIP TC6 Third International Workshop on Protocol Test Systems , 1991 .

[9]  Paul D. Amer,et al.  Improvements on UIO Sequence Generation and Partial UIO Sequences , 1992, PSTV.

[10]  Son T. Vuong,et al.  TESTGEN+: An Integrated Environment for Protocol Test Suite Generation, Selection and Validation , 1993, FORTE.

[11]  Son T. Vuong,et al.  A Metric Based Theory of Test Selection and Coverage , 1993, PSTV.

[12]  Qin Li,et al.  The UBC Protocol Testing Environment , 1993, Protocol Test Systems.

[13]  R. Lai,et al.  How could research on testing of communicating systems become more industrially relevant , 1996 .

[14]  J. A. Bondy,et al.  Graph Theory with Applications , 1978 .

[15]  Xiaojun Shen,et al.  A new protocol conformance test generation method and experimental results , 1992, SAC '92.

[16]  Krishan K. Sabnani,et al.  An experience in estimating fault coverage of a protocol test , 1988, IEEE INFOCOM '88,Seventh Annual Joint Conference of the IEEE Computer and Communcations Societies. Networks: Evolution or Revolution?.

[17]  Ferhat Khendek,et al.  Test Selection Based on Finite State Models , 1991, IEEE Trans. Software Eng..

[18]  Deepinder P. Sidhu,et al.  Formal Methods for Protocol Testing: A Detailed Study , 1989, IEEE Trans. Software Eng..

[19]  Fabrizio Lombardi,et al.  Evaluation and improvement of fault coverage of conformance testing by UIO sequences , 1992, IEEE Trans. Commun..

[20]  Deepinder P. Sidhu,et al.  Fault coverage of probabilistic test sequences , 1990 .

[21]  Richard J. Linn,et al.  Producing Tests for Implementations of OSI Protocols , 1983, Protocol Specification, Testing, and Verification.

[22]  S.T. Vuong,et al.  TESTGEN: An environment for protocol test suite generation and selection , 1994, Comput. Commun..

[23]  Tadanori Mizuno,et al.  Protocol Test Systems , 1995, IFIP — The International Federation for Information Processing.

[24]  Gregor von Bochmann,et al.  TESTL: An Environment for Incremental Test Suite Design Based on Finite-State Models , 1991, Protocol Test Systems.

[25]  Katalin Tarnay,et al.  Testing of Communicating Systems , 1999, IFIP — The International Federation for Information Processing.

[26]  Fabrizio Lombardi,et al.  Protocol Conformance Testing by Discriminating UIO Sequences , 1991, PSTV.

[27]  Krishan K. Sabnani,et al.  Algorithmic generation of protocol conformance tests , 1990, AT&T Technical Journal.

[28]  Taghi M. Khoshgoftaar,et al.  Early Quality Prediction: A Case Study in Telecommunications , 1996, IEEE Softw..

[29]  Colin H. West,et al.  Experiences with a Random Test Driver , 1995, Comput. Networks ISDN Syst..

[30]  Hendrik Brinksma,et al.  Protocol Test Systems, IV. , 1992 .

[31]  Hartmut König,et al.  SELEXPERT - A Knowledge-based Tool for Test Case Selection , 1995, FORTE.

[32]  Samuel T. Chanson,et al.  On Ferry Clip Approaches in Protocol Testing , 1989, Comput. Networks.

[33]  C.A. Vissers,et al.  Formal description techniques , 1983, Proceedings of the IEEE.

[34]  Parag K. Lala,et al.  Fault tolerant and fault testable hardware design , 1985 .

[35]  D. Rayner.  OSI Conformance Testing , 1987, Comput. Networks.

[36]  Rudolf Jan Heijink.  FAITH, a General Purpose Protocol Test System for ISDN , 1994, Comput. Networks ISDN Syst..

[37]  Gregor von Bochmann,et al.  Synchronization and Specification Issues in Protocol Testing , 1984, IEEE Trans. Commun..

[38]  Tadanori Mizuno,et al.  TENT: Test Sequence Generation Tool for Communication Systems , 1989, FORTE.

[39]  Jan Kroon,et al.  Interactive Test Generation from LOTOS Specifications , 1992, Protocol Test Systems.

[40]  Per Gunningberg,et al.  Experiences with Estelle, LOTOS and SDL: A Protocol Implementation Experiment , 1991, Comput. Networks ISDN Syst..

[41]  B. Svrcek,et al.  Proceedings of the 1990 Summer computer simulation conference , 1990 .

[42]  Malati Hedge,et al.  An Experience in Using UIO Sequences and Rural Chinese Postman Tours of Conformance Testing of a Q.931 Implementation , 1992, NETWORKS.

[43]  Gregor von Bochmann,et al.  Testing Non-Deterministic State Machines with Fault Coverage , 1991, Protocol Test Systems.

[44]  Fabrizio Lombardi,et al.  Protocol conformance testing using multiple UIO sequences , 1989, IEEE Trans. Commun..

[45]  G. Bochmann,et al.  Fault Models in Testing , 1991, Protocol Test Systems.

[46]  Sanjoy Paul,et al.  Generating maximal fault coverage conformance test sequences of reduced length for communication protocols , 1993, 1993 International Conference on Network Protocols.

[47]  Sanjoy Paul,et al.  Generating minimal length test sequences for conformance testing of communication protocols , 1991, IEEE INFCOM '91. The conference on Computer Communications. Tenth Annual Joint Comference of the IEEE Computer and Communications Societies Proceedings.

[48]  Güney Gönenç,et al.  A Method for the Design of Fault Detection Experiments , 1970 .

[49]  Anthony Wiles,et al.  Experiences with Computer Aided Test Suite Generation , 1993, Protocol Test Systems.

[50]  Gregor von Bochmann,et al.  Error detection with multiple observers , 1985, PSTV.

[51]  Behçet Sarikaya,et al.  Estelle-based test generation tool , 1991, Comput. Commun..

[52]  Richard Lai,et al.  Industrial and Academic Protocol Testing: the Gap and the Means of Convergence , 1995, Comput. Networks ISDN Syst..

[53]  O. Rafiq.  From Conformance Testing to Interoperability Testing , 1990 .

[54]  S. V. Raghavan,et al.  Experience in Test Generation Using Multi-Level Approach , 1995, Comput. Networks ISDN Syst..

[55]  Krishan K. Sabnani,et al.  A Protocol Test Generation Procedure , 1988, Comput. Networks.

[56]  Sanjoy Paul,et al.  Structural analysis of protocol specifications and generation of maximal fault coverage conformance test sequences , 1994, TNET.

[57]  Carl A. Sunshine,et al.  Protocol Specification, Testing and Verification - Guest Editorial , 1982, Comput. Networks.

[58]  Deepinder P. Sidhu,et al.  On sufficient conditions for an efficient protocol conformance test generation technique based on rural Chinese postman problem , 1991 .

[59]  Niklaus Wirth,et al.  Programmieren in Modula-2 , 1985 .

[60]  Yanghee Choi,et al.  Approaches utilizing segment overlap to minimize test sequences , 1990, PSTV.

[61]  Juhani Malka,et al.  Performance Characteristics of Portable ISO/OSI Protocol Implementations , 1992, INDC.