Applying simulation and design of experiments to the embedded software testing process

This paper presents some original solutions with regard to the deployment of the U.S. Department of Defense Simulation, Test and Evaluation Process (DoD STEP), using an automated target tracking radar system as a case study. Besides the integration of modelling and simulation, to form a model‐based approach to the software testing process, the number of experiments, i.e. test cases, have been dramatically reduced by applying an optimized design‐of‐experiment plan and an orthogonal array‐based robust testing methodology. Also, computer‐based simulation at various abstraction levels of the system/software under test can serve as a test oracle. Simulation‐based (stochastic) experiments, combined with optimized design‐of‐experiment plans, in the case study have shown a minimum productivity increase of 100 times in comparison to current practice without DoD STEP deployment. Copyright © 2004 John Wiley & Sons, Ltd.

[1]  J. S. Hunter,et al.  Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. , 1979 .

[2]  Robert G. Sargent,et al.  Validation and verification of simulation models , 1999, Proceedings of the 2004 Winter Simulation Conference, 2004..

[3]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[4]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[5]  D. Stamatis,et al.  Orthogonal Arrays and Linear Graphs , 2002 .

[6]  Jai Menon,et al.  A distributed development environment for embedded software , 1993, Softw. Pract. Exp..

[7]  Robert G. Sargent,et al.  Verification, validation and accreditation of simulation models , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[8]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[9]  Kai-Yuan Cai,et al.  Optimal software testing and adaptive software testing in the context of software cybernetics , 2002, Inf. Softw. Technol..

[10]  Robert Brownlie,et al.  Robust testing of AT&T PMX/StarMAIL using OATS , 1992, AT&T Technical Journal.

[11]  Petru Eles,et al.  An approach to incremental design of distributed embedded systems , 2001, Proceedings of the 38th Design Automation Conference (IEEE Cat. No.01CH37232).

[12]  K. Reinholtz.  Applying simulation to the development of spacecraft flight software , 1999, 1999 IEEE Aerospace Conference. Proceedings (Cat. No.99TH8403).

[13]  John Reeves,et al.  A Test Planning Methodology From Concept Development Through Test Execution , 1999 .

[14]  Alan M. Christie.  Simulation: An Enabling Technology in Software Engineering , 1999 .

[15]  Franz Lehner,et al.  Requirements Engineering as a Success Factor in Software Projects , 2001, IEEE Softw..

[16]  Debra J. Richardson,et al.  TAOS: Testing with Analysis and Oracle Support , 1994, ISSTA '94.

[17]  Capers Jones,et al.  Applied software measurement: assuring productivity and quality , 1991 .

[18]  A. Alan B. Pritsker,et al.  Introduction to simulation and SLAM II , 1979 .

[19]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[20]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[21]  Taratip Suwannasart,et al.  Developing a Testing Maturity Model, Part II , 1996 .

[22]  Don Caughlin.  An integrated approach to verification, validation, and accredition of models and simulations , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[23]  David Chapman.  A program testing assistant , 1982, CACM.