Software and Hardware Testing Using Combinatorial Covering Suites

In the 21st century our society is becoming more and more dependent on software systems. The safety of these systems and the quality of our lives is increasingly dependent on the quality of such systems. A key element in the manufacture and quality assurance process in software engineering is the testing of software and hardware systems. The construction of efficient combinatorial covering suites has important applications in the testing of hardware and software. In this paper we define the general problem, discuss the lower bounds on the size of covering suites, and give a series of constructions that achieve these bounds asymptotically. These constructions include the use of finite field theory, extremal set theory, group theory, coding theory, combinatorial recursive techniques, and other areas of computer science and mathematics. The study of these combinatorial covering suites is a fascinating example of the interplay between pure mathematics and the applied problems generated by software and hardware engineers. The wide range of mathematical techniques used, and the often unexpected applications of combinatorial covering suites make for a rewarding study.

[1]  R. Fisher.  An examination of the different possible solutions of a problem in incomplete blocks. , 1940 .

[2]  K. A. Bush.  A Generalization of a Theorem due to MacNeish , 1952 .

[3]  K. A. Bush.  Orthogonal Arrays of Index Unity , 1952 .

[4]  Paul Erdös,et al.  On the Maximal Number of Pairwise Orthogonal Latin Squares of a Given Order , 1960, Canadian Journal of Mathematics.

[5]  R. C. Bose,et al.  Further Results on the Construction of Mutually Orthogonal Latin Squares and the Falsity of Euler's Conjecture , 1960, Canadian Journal of Mathematics.

[6]  P. Erdös,et al.  INTERSECTION THEOREMS FOR SYSTEMS OF FINITE SETS , 1961 .

[7]  Joel H. Spencer,et al.  Families of k-independent sets , 1973, Discret. Math..

[8]  G. Katona.  Two applications (for search theory and truth functions) of Sperner type theorems , 1973 .

[9]  Curtis Greene.  Sperner Families and Partitions of A Partially Ordered Set , 1975 .

[10]  Arnfried Kemnitz,et al.  Calculations for Bertrand's Postulate , 1981 .

[11]  Donald T. Tang,et al.  Exhaustive Test Pattern Generation with Constant Weight Vectors , 1983, IEEE Transactions on Computers.

[12]  Donald T. Tang,et al.  Iterative Exhaustive Pattern Generation for Logic Testing , 1984, IBM J. Res. Dev..

[13]  G. Roux.  K-proprietes dans des tableaux de n colonnes : cas particulier de la k-surjectivite et de la k-permutivite , 1987 .

[14]  Gadiel Seroussi,et al.  Vector sets for exhaustive testing of logic circuits , 1988, IEEE Trans. Inf. Theory.

[15]  Moni Naor,et al.  Small-bias probability spaces: efficient constructions and applications , 1990, STOC '90.

[16]  Klaus Metsch,et al.  Improvement of Bruck's completion theorem , 1991, Des. Codes Cryptogr..

[17]  I. S. Grunskii,et al.  Recursive generation of locally complete tests , 1992 .

[18]  N. Sloane.  Covering arrays and intersecting codes , 1993 .

[19]  Nancy G. Leveson,et al.  An investigation of the Therac-25 accidents , 1993, Computer.

[20]  Luisa Gargano,et al.  Sperner capacities , 1993, Graphs Comb..

[21]  Luisa Gargano,et al.  Capacities: From Information Theory to Extremal Set Theory , 1994, J. Comb. Theory, Ser. A.

[22]  D. West.  Introduction to Graph Theory , 1995 .

[23]  Aravind Srinivasan,et al.  Splitters and near-optimal derandomization , 1995, Proceedings of IEEE 36th Annual Foundations of Computer Science.

[24]  C. Colbourn,et al.  CRC Handbook of Combinatorial Designs , 1996 .

[25]  J. L. Lions.  ARIANE 5 Flight 501 Failure: Report by the Enquiry Board , 1996 .

[26]  S. Alpern,et al.  Minimax Rendezvous on the Line , 1996 .

[27]  Anant P. Godbole,et al.  t-Covering Arrays: Upper Bounds and Poisson Approximations , 1996, Combinatorics, Probability and Computing.

[28]  Alan Edelman,et al.  The Mathematics of the Pentium Division Bug , 1997, SIAM Rev..

[29]  Kathryn Fraughnaugh,et al.  Introduction to graph theory , 1973, Mathematical Gazette.

[30]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[31]  Lucia Moura,et al.  Lower Bounds for Transversal Covers , 1998, Des. Codes Cryptogr..

[32]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[33]  Yossi Azar,et al.  Approximating Probability Distributions Using Small Sample Spaces , 1998, Comb..

[34]  Charles J. Colbourn,et al.  Covering Arrays of Strength Three , 1999, Des. Codes Cryptogr..

[35]  Shmuel Gal.  Rendezvous Search on the Line , 1999, Oper. Res..

[36]  Charles J. Colbourn,et al.  Covering arrays of strength 3 , 1999 .

[37]  Alan W. Williams,et al.  Determination of Test Configurations for Pair-Wise Interaction Coverage , 2000, TestCom.

[38]  Brett Stevens,et al.  A direct construction of transversal covers using group divisible designs , 2002, Ars Comb..

[39]  Donald L. Kreher,et al.  On the state of strength‐three covering arrays , 2002 .

[40]  Alan Hartman,et al.  Using a model-based test generator to test for standard conformance , 2002, IBM Syst. J..

[41]  Yu Lei,et al.  A Test Generation Strategy for Pairwise Testing , 2002, IEEE Trans. Software Eng..

[42]  Kari J. Nurmela,et al.  Upper bounds for covering arrays by tabu search , 2004, Discret. Appl. Math..

[43]  Shmuel Zaks,et al.  On sets of Boolean n-vectors with all k-projections surjective , 2004, Acta Informatica.

[44]  Eduardo Valido-Cabrera.  Software reliability methods , 2006 .