Zero-Knowledge Blackbox Testing: where are the Faults?

We derive a quantitative relationship between the maximal entropy rate achieved by a blackbox software system's specification graph, and the probability of faults Pn obtained by testing the system, as a function of the length n of a test sequence. By equating “blackbox” to the maximal entropy principle, we model the specification graph as a Markov chain that, for each distinct value of n, achieves the maximal entropy rate for that n. Hence the Markov transition probability matrices are not constant in n, but form a sequence of transition matrices T1,…, Tn. We prove that, for nontrivial specification graphs, the probability of finding faults goes asymptotically to zero as the test length n increases, regardless of the evolution of Tn. This implies that zero-knowledge testing is practical only for small n. We illustrate the result using a concrete example of a system specification graph for an autopilot control system, and plot its curve Pn.

[1]  Margus Veanes,et al.  Optimal strategies for testing nondeterministic systems , 2004, ISSTA '04.

[2]  Mike Holcombe,et al.  Statechart testing method for aircraft control systems , 2001, Softw. Test. Verification Reliab..

[3]  Z. Burda,et al.  Localization of the maximal entropy random walk. , 2008, Physical review letters.

[4]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[5]  E. Jaynes Probability theory : the logic of science , 2003 .

[6]  Andy Podgurski,et al.  The Probabilistic Program Dependence Graph and Its Application to Fault Diagnosis , 2008, IEEE Transactions on Software Engineering.

[7]  Zhe Dang,et al.  Information gain of black-box testing , 2011, Formal Aspects of Computing.

[8]  Hongyu Zhang On the Distribution of Software Faults , 2008, IEEE Transactions on Software Engineering.

[9]  L. B. Boza Asymptotically Optimal Tests for Finite Markov Chains , 1971 .

[10]  Per Runeson,et al.  A Second Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems , 2007, IEEE Transactions on Software Engineering.

[11]  Imre Csisźar,et al.  The Method of Types , 1998, IEEE Trans. Inf. Theory.

[12]  Zhe Dang,et al.  Typical Paths of a Graph , 2011, Fundam. Informaticae.

[13]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[14]  Norman E. Fenton,et al.  Quantitative Analysis of Faults and Failures in a Complex Software System , 2000, IEEE Trans. Software Eng..