Entropy and software systems: towards an information-theoretic foundation of software testing

We integrate information theory into software testing. In particular, we use entropy in information theory to measure the amount of uncertainty in a software system before it is fully tested, and we show how the amount decreases when we test the system. Moreover, we introduce behaviorial complexity as a novel complexity metric for labeled graphs (which can be interpreted as control flow graphs, design specifications, etc.), which is also based on information theory. We seek practical approaches in testing real systems using the above theories, and we apply our novel approaches in testing model-based embedded systems and network intrusion detection systems. Our information-theoretic approach is syntax-independent, which is a desired property in software testing.

[1]  Oscar H. Ibarra,et al.  Bond computing systems: a biologically inspired and high-level dynamics model for pervasive computing , 2009, Natural Computing.

[2]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[3]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems, Advanced Lectures , 2005 .

[4]  A. Ahmad.  Measuring software complexity , 1981 .

[5]  Insup Lee,et al.  Specification-based testing with linear temporal logic , 2004, Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, 2004. IRI 2004..

[6]  Joseph K. Kearney,et al.  Software complexity measurement , 1986, CACM.

[7]  Kuo-Chung Tai.  A program complexity metric based on data flow information in control graphs , 1984, ICSE '84.

[8]  Nina Yevtushenko,et al.  Testing Transition Systems with Input and Output Testers , 2003, TestCom.

[9]  Jan Tretmans,et al.  TorX: Automated Model-Based Testing , 2003 .

[10]  Jan Tretmans,et al.  Model Based Testing with Labelled Transition Systems , 2008, Formal Methods and Testing.

[11]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[12]  Amir Pnueli,et al.  On the Development of Reactive Systems , 1989, Logics and Models of Concurrent Systems.

[13]  Jitender Kumar Chhabra,et al.  Evaluation of object-oriented spatial complexity measures , 2009, SOEN.

[14]  Roy M. Howard,et al.  Linear System Theory , 1992 .

[15]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[16]  O. F. Cook.  The Method of Types , 1898 .

[17]  Sigrid Eldh.  Software Testing Techniques , 2007 .

[18]  Mihalis Yannakakis,et al.  Black Box Checking , 1999, FORTE.

[19]  Paul J. Layzell,et al.  Spatial measures of software complexity , 1999, PPIG.

[20]  Zhe Dang,et al.  Information gain of black-box testing , 2011, Formal Aspects of Computing.

[21]  Yong Wang,et al.  Automata on Multisets of Communicating Objects , 2008, UC.

[22]  Richard E. Blahut,et al.  Principles and practice of information theory , 1987 .

[23]  George A. Miller,et al.  Finite State Languages , 1958, Inf. Control..

[24]  Wilfred J. Hansen,et al.  Measurement of program complexity by the pair: (Cyclomatic Number, Operator Count) , 1978, SIGP.

[25]  M. Minsky.  Recursive Unsolvability of Post's Problem of "Tag" and other Topics in Theory of Turing Machines , 1961 .

[26]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[27]  Oscar H. Ibarra,et al.  On Stateless Automata and P Systems , 2008, Int. J. Found. Comput. Sci..

[28]  L. B. Boza.  Asymptotically Optimal Tests for Finite Markov Chains , 1971 .

[29]  Mary Jean Harrold,et al.  Testing: a roadmap , 2000, ICSE '00.

[30]  Hong Zhu,et al.  Test data adequacy measurement , 1993, Softw. Eng. J..

[31]  Eitan M. Gurari,et al.  Introduction to the theory of computation , 1989 .

[32]  Abraham Kandel,et al.  The data mining approach to automated software testing , 2003, KDD '03.

[33]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[34]  Mahadev Satyanarayanan,et al.  Pervasive computing: vision and challenges , 2001, IEEE Wirel. Commun..

[35]  Warren Harrison,et al.  An Entropy-Based Measure of Software Complexity , 1992, IEEE Trans. Software Eng..

[36]  James Ladyman,et al.  Physics and Computation: The Status of Landauer's Principle , 2007, CiE.

[37]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[38]  J. Miller.  Numerical Analysis , 1966, Nature.

[39]  E. Schrödinger.  Die gegenwärtige Situation in der Quantenmechanik , 1935, Naturwissenschaften.

[40]  Oscar H. Ibarra,et al.  On Model-Checking of P Systems , 2005, UC.

[41]  Byoung-Seon Choi,et al.  Conditional limit theorems under Markov conditioning , 1987, IEEE Trans. Inf. Theory.

[42]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[43]  Vijay Kumar,et al.  Hierarchical modeling and analysis of embedded systems , 2003, Proc. IEEE.

[44]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[45]  Yingxu Wang,et al.  A new measure of software complexity based on cognitive weights , 2003, Canadian Journal of Electrical and Computer Engineering.

[46]  R. Landauer,et al.  Irreversibility and heat generation in the computing process , 1961, IBM J. Res. Dev..

[47]  Robin Milner,et al.  A Calculus of Communicating Systems , 1980, Lecture Notes in Computer Science.

[48]  Min Zhang,et al.  Performance Comparison of Software Complexity Metrics in an Open Source Project , 2007, EuroSPI.

[49]  S. L. Gerhart,et al.  Toward a theory of test data selection , 1975, IEEE Transactions on Software Engineering.

[50]  Margus Veanes,et al.  Optimal strategies for testing nondeterministic systems , 2004, ISSTA '04.

[51]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[52]  Sudipto Ghosh,et al.  Issues in Testing Distributed Component-Based Systems , 1999 .

[53]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[54]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[55]  Gaoyan Xie,et al.  Testing Systems of Concurrent Black-Boxes-An Automata-Theoretic and Decompositional Approach , 2005, FATES.

[56]  Nancy A. Lynch,et al.  An introduction to input/output automata , 1989 .

[57]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[58]  K. K. Aggarwal,et al.  Code and data spatial complexity: two important software understandability measures , 2003, Inf. Softw. Technol..

[59]  A. Jefferson Offutt,et al.  Coverage criteria for logical expressions , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[60]  M. W. Shields.  An Introduction to Automata Theory , 1988 .

[61]  Maurice H. Halstead,et al.  Elements of software science (Operating and programming systems series) , 1977 .

[62]  Constance L. Heitmeyer,et al.  Software Cost Reduction , 2002 .

[63]  Gheorghe Paun,et al.  Computing with Membranes , 2000, J. Comput. Syst. Sci..

[64]  Arkadiusz Wojna,et al.  Counter Machines , 1999, Inf. Process. Lett..

[65]  Boris Beizer,et al.  Black Box Testing: Techniques for Functional Testing of Software and Systems , 1996, IEEE Software.

[66]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[67]  Martin R. Woodward,et al.  A Measure of Control Flow Complexity in Program Text , 1979, IEEE Transactions on Software Engineering.

[68]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[69]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[70]  Yong Wang,et al.  Automata and processes on multisets of communicating objects , 2010, Natural Computing.

[71]  Jan A. Bergstra,et al.  Algebra of Communicating Processes with Abstraction , 1985, Theor. Comput. Sci..