Information gain of black-box testing

For model-based black-box testing, test cases are often selected from the syntactic appearance of the specification of the system under test, according to a pre-given test data adequacy criterion. We introduce a novel approach that is semantics-based, independent of the syntactic appearance of the system specification. Basically, we model the system under test as a random variable, whose sample space consists of all possible behavior sets (with respect to the specification) over the known interface of the black-box. The entropy of the system is measured as the (Shannon) entropy of the random variable. In our criterion, the coverage of a test set is measured as the expected amount of entropy decrease (i.e. the expected amount of information gained) once the test set is run. Since our criterion is syntactic independent, we study the notion of information-optimal software testing where, within a given constraint, a test set is selected to gain the most information.

[1]  I. H. Öğüş,et al.  NATO ASI Series , 1997 .

[2]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[3]  A. Jefferson Offutt,et al.  Coverage criteria for logical expressions , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[4]  Jan Tretmans,et al.  TorX: Automated Model-Based Testing , 2003 .

[5]  Gaoyan Xie,et al.  Fundamental studies on a decompositional and hybrid approach to automatic verification of component-based systems , 2005 .

[6]  Zhe Dang,et al.  Optimal Software Testing - A Cooling Down Process , 2009, FCS.

[7]  Jan Tretmans,et al.  Model Based Testing with Labelled Transition Systems , 2008, Formal Methods and Testing.

[8]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[9]  Rolf Landauer,et al.  Irreversibility and heat generation in the computing process , 1961, IBM J. Res. Dev..

[10]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[11]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[12]  Nina Yevtushenko,et al.  Testing Transition Systems with Input and Output Testers , 2003, TestCom.

[13]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[14]  Amir Pnueli,et al.  On the Development of Reactive Systems , 1989, Logics and Models of Concurrent Systems.

[15]  Boris Beizer,et al.  Black Box Testing: Techniques for Functional Testing of Software and Systems , 1996, IEEE Software.

[16]  Hong Zhu,et al.  Test data adequacy measurement , 1993, Softw. Eng. J..

[17]  Abraham Kandel,et al.  The data mining approach to automated software testing , 2003, KDD '03.

[18]  Mihalis Yannakakis,et al.  Black Box Checking , 1999, FORTE.

[19]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[20]  Insup Lee,et al.  Specification-based testing with linear temporal logic , 2004, Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, 2004. IRI 2004..

[21]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[22]  John J. Marciniak,et al.  Encyclopedia of Software Engineering , 1994, Encyclopedia of Software Engineering.

[23]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[24]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[25]  Fabien Peureux,et al.  I/O-automata Based Testing , 2004, Model-Based Testing of Reactive Systems.

[26]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[27]  James Ladyman,et al.  Physics and Computation: The Status of Landauer's Principle , 2007, CiE.

[28]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems, Advanced Lectures , 2005 .

[29]  Ismael Rodríguez,et al.  Formally transforming user-model testing problems into implementer-model testing problems and viceversa , 2009, J. Log. Algebraic Methods Program..

[30]  Constance L. Heitmeyer,et al.  Software Cost Reduction , 2002 .

[31]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[32]  S. L. Gerhart,et al.  Toward a theory of test data selection , 1975, IEEE Transactions on Software Engineering.

[33]  Margus Veanes,et al.  Optimal strategies for testing nondeterministic systems , 2004, ISSTA '04.

[34]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[35]  Gaoyan Xie,et al.  Testing Systems of Concurrent Black-Boxes-An Automata-Theoretic and Decompositional Approach , 2005, FATES.

[36]  Nancy A. Lynch,et al.  An introduction to input/output automata , 1989 .

[37]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..