Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing

Many industrial products are based on the use of embedded computer systems. Usually, these systems have to fulfil real-time requirements, and correct system functionality depends on their logical correctness as well as on their temporal correctness. In order to verify the temporal behavior of real-time systems, previous scientific work has, to a large extent, concentrated on static analysis techniques. Although these techniques offer the possibilty of providing safe estimates of temporal behavior for certain cases, there are a number of cases in practice for which static analysis can not be easily applied. Furthermore, no commercial tools for timing analysis of real-world programs are available. Therefore, the developed systems have to be thoroughly tested in order to detect existing deficiencies in temporal behavior, as well as to strengthen the confidence in temporal correctness. An investigation of existing test methods shows that they mostly concentrate on testing for logical correctness. They are not specialised in the examination of temporal correctness which is also essential to real-time systems. For this reason, existing test procedures must be supplemented by new methods which concentrate on determining whether the system violates its specified timing constraints. Normally, a violation means that outputs are produced too early, or their computation takes too long. The task of the tester therefore is to find the input situations with the longest or shortest execution times, in order to check whether they produce a temporal error. If the search for such inputs is interpreted as a problem of optimization, evolutionary computation can be used to automatically find the inputs with the longest or shortest execution times. This automatic search for accurate test data by means of evolutionary computation is called evolutionary testing. Experiments using evolutionary testing on a number of programs with up to 1511 LOC and 5000 input parameters have successfully identified new longer and shorter execution times than had been found using other testing techniques. Evolutionary testing, therefore, seems to be a promising approach for the verification of timing constraints. A combination of evolutionary testing and systematic testing offers further opportunities to improve the test quality, and could lead to an effective test strategy for real-time systems.

[1]  Michael A. Hennell,et al.  Automated Testing Techniques for Real-Time Embedded Software , 1987, ESEC.

[2]  William C. Hetzel,et al.  The complete guide to software testing , 1984 .

[3]  Edward B. Boden,et al.  Testing software using order-based genetic algorithms , 1996 .

[4]  Chang Yun Park,et al.  Predicting program execution times by analyzing static and dynamic program paths , 1993, Real-Time Systems.

[5]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[6]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[7]  Xiao Jun.  Application of Genetic Algorithms , 2005 .

[8]  John J. Grefenstette,et al.  Test and evaluation by genetic algorithms , 1993, IEEE Expert.

[9]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[10]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[11]  Miguel Felder,et al.  Testing Timing Behavior of Real-Time Software , 1997 .

[12]  Marc Roper.  Computer aided software testing using genetic algorithms , 1997 .

[13]  B. Dasarathy,et al.  Timing Constraints of Real-Time Systems: Constructs for Expressing Them, Methods of Validating Them , 1989, IEEE Transactions on Software Engineering.

[14]  Peter P. Puschner,et al.  Problems in Static Worst-Case Execution Time Analysis , 1997, MMB.

[15]  Joachim Wegener,et al.  A Comparison of Static Analysis and Evolutionary Testing for the Verification of Timing Constraints , 2004, Real-Time Systems.

[16]  John A. Stankovic,et al.  Misconceptions About Real-Time Computing , 1988, Computer.

[17]  James E. Baker,et al.  Reducing Bias and Inefficienry in the Selection Algorithm , 1987, ICGA.

[18]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[19]  Walter S. Heath.  Real-Time Software Techniques , 1990 .

[20]  Bryan F. Jones,et al.  Automatic structural testing using genetic algorithms , 1996, Softw. Eng. J..

[21]  David B. Whalley,et al.  Integrating the timing analysis of pipelining and instruction caching , 1995, Proceedings 16th IEEE Real-Time Systems Symposium.

[22]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[23]  Joachim Wegener,et al.  A Comparison of Static Analysis and Evolutionary Testing for the Verification of Timing Constraints , 1998, Proceedings. Fourth IEEE Real-Time Technology and Applications Symposium (Cat. No.98TB100245).

[24]  Sandro Morasca,et al.  FUNCTIONAL TEST CASE GENERATION FOR REAL-TIME SYSTEMS , 1993 .

[25]  Joachim Wegener,et al.  Testing real-time systems using genetic algorithms , 1997, Software Quality Journal.

[26]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[27]  Frank Müller,et al.  Generalizing timing predictions to set-associative caches , 1997, Proceedings Ninth Euromicro Workshop on Real Time Systems.

[28]  Matthias Grochtmann,et al.  Test Case Design Using Classification Trees and the Classification-Tree Editor CTE , 1995 .

[29]  Harmen-Hinrich Sthamer,et al.  The automatic generation of software test data using genetic algorithms , 1995 .

[30]  Duncan Clarke,et al.  Testing Real-Time Constraints in a Process Algebraic Setting , 1995, 1995 17th International Conference on Software Engineering.

[31]  Sharad Malik,et al.  Cache modeling for real-time software: beyond direct mapped instruction caches , 1996, 17th IEEE Real-Time Systems Symposium.

[32]  Sang Lyul Min,et al.  Efficient worst case timing analysis of data caching , 1996, Proceedings Real-Time Technology and Applications.

[33]  Matthias Grochtmann,et al.  Classification trees for partition testing , 1993, Softw. Test. Verification Reliab..

[34]  Matthias Grochtmann,et al.  - Yet Another Computer-Aided Software Testing Tool? , 1994 .

[35]  Duncan Clarke,et al.  Testing real-time constraints , 1996 .