Integrated intelligent modeling, simulation and design of experiments for software testing process

This paper presents some original solutions with regard to deployment of the US Department of Defense Simulation, Test and Evaluation Process (DoD STEP), using an automated target tracking radar system as a case study of Integrated and Optimized Software Testing Process (IOSTP) framework deployment. This paper is a composite of what is in hand and within reasonable reach in the application of many science and engineering areas that can be successfully applied to software projects management to assure stable (observable and controllable) development process. Besides the integration of modeling and simulation, to form a model-based approach to the software testing process, the number of experiments, i.e. test cases, have been dramatically reduced by applying an optimized design-of-experiment plan and an orthogonal array-based robust testing methodology. Intelligence is gathered throughout the project lifecycle, from early reviews to final acceptance testing. The intelligence gathered, enables project and stakeholder management to judge how and whether progress is being made. Computer-based simulation at various abstraction levels of the system/software under test can serve as a test oracle too. Simulation-based (stochastic) experiments, combined with optimized design-of-experiment plans, in the case study, have shown a minimum productivity increase of 100 times in comparison to current practice without IOSTP deployment.

[1]  A. Alan B. Pritsker,et al.  Introduction to simulation and SLAM II , 1979 .

[2]  Alan M. Christie.  Simulation: An Enabling Technology in Software Engineering , 1999 .

[3]  Lj Lazić,et al.  Applying simulation and design of experiments to the embedded software testing process: Research Articles , 2004 .

[4]  R. L. Erickson,et al.  Improved quality of protocol testing through techniques of experimental design , 1994, Proceedings of ICC/SUPERCOMM'94 - 1994 International Conference on Communications.

[5]  Petru Eles,et al.  An approach to incremental design of distributed embedded systems , 2001, Proceedings of the 38th Design Automation Conference (IEEE Cat. No.01CH37232).

[6]  C. L. Mallows,et al.  Applying Design of Experiments to Software Testing , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[7]  Capers Jones,et al.  Applied software measurement (2nd ed.): assuring productivity and quality , 1996 .

[8]  Robert G. Sargent,et al.  Verification, validation, and accreditation: verification, validation, and accreditation of simulation models , 2000, WSC '00.

[9]  Dusan M. Velasevic,et al.  Applying simulation and design of experiments to the embedded software testing process , 2004, Softw. Test. Verification Reliab..

[10]  Capers Jones,et al.  Applied software measurement: assuring productivity and quality , 1991 .

[11]  Robert Brownlie,et al.  Robust testing of AT&T PMX/StarMAIL using OATS , 1992, AT&T Technical Journal.

[12]  Nikos E. Mastorakis,et al.  RBOSTP: risk-based optimization of software testing process part 1 , 2005 .

[13]  A. Alan B. Pritsker.  Introduction to simulation and SLAM II (2nd ed.) , 1984 .

[14]  Jai Menon,et al.  A distributed development environment for embedded software , 1993, Softw. Pract. Exp..

[15]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[16]  Nikos E. Mastorakis,et al.  The use of modeling & simulation-based analysis & optimization of software testing , 2005 .

[17]  Ieee Std,et al.  IEEE Standard for Software Verification and Validation , 2008 .

[18]  Nikos E. Mastorakis,et al.  The COTECOMO: COnstractive Test Effort COst MOdel , 2009 .

[19]  Robert G. Sargent,et al.  Validation and verification of simulation models , 1999, Proceedings of the 2004 Winter Simulation Conference, 2004..

[20]  Nikos E. Mastorakis,et al.  Techniques to reduce a set of test cases , 2006 .

[21]  Alan J. Mayne,et al.  Introduction to Simulation and SLAM , 1979 .

[22]  C. L. Mallows,et al.  Applying design of experiments to software testing: experience report , 1997, ICSE '97.