Support planning and controlling of early quality assurance by combining expert judgment and defect data—a case study

Planning quality assurance (QA) activities in a systematic way and controlling their execution are challenging tasks for companies that develop software or software-intensive systems. Both require estimation capabilities regarding the effectiveness of the applied QA techniques and the defect content of the checked artifacts. Existing approaches for these purposes need extensive measurement data from historical projects. Due to the fact that many companies do not collect enough data for applying these approaches (especially for the early project lifecycle), they typically base their QA planning and controlling solely on expert opinion. This article presents a hybrid method combining commonly available measurement data and context-specific expert knowledge. To evaluate the method’s applicability and usefulness, we conducted a case study in the context of independent verification and validation activities for critical software in the space domain. A hybrid defect content and effectiveness model was developed for the software requirements analysis phase and evaluated with available legacy data. One major result is that the hybrid model provides improved estimation accuracy when compared to applicable models based solely on data. The mean magnitude of relative error (MMRE) determined by cross-validation is 29.6% compared to 76.5% obtained by the most accurate data-based model.

[1]  Martin R. Woodward.  Editorial: Little bugs and big bugs , 2002, Softw. Test. Verification Reliab..

[2]  Jürgen Münch,et al.  State of the Practice in Software Effort Estimation: A Survey and Literature Review , 2008, CEE-SET.

[3]  Khaled El Emam,et al.  The application of subjective estimates of effectiveness to controlling software inspections , 2000, J. Syst. Softw..

[4]  Michael Eichberg,et al.  A Handbook of Software and Systems Engineering , 2009 .

[5]  Kenji Yokoyama,et al.  Development of a hybrid cost estimation model in an iterative manner , 2006, ICSE.

[6]  L. Delbeke.  Quasi-experimentation - design and analysis issues for field settings - cook,td, campbell,dt , 1980 .

[7]  D. Ross Jeffery,et al.  Cost estimation for web applications , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[8]  Jane M. Booker,et al.  Eliciting and analyzing expert judgement - a practical guide , 2001, ASA-SIAM series on statistics and applied probability.

[9]  Maurice H. Halstead,et al.  Elements of software science (Operating and programming systems series) , 1977 .

[10]  Natalia Juristo Juzgado,et al.  A survey on testing technique empirical studies: how limited is our knowledge , 2002, Proceedings International Symposium on Empirical Software Engineering.

[11]  Maurice H. Halstead,et al.  Elements of software science , 1977 .

[12]  Norman E. Fenton,et al.  A Critique of Software Defect Prediction Models , 1999, IEEE Trans. Software Eng..

[13]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[14]  Stephen G. MacDonell,et al.  What accuracy statistics really measure , 2001, IEE Proc. Softw..

[15]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[16]  Lionel C. Briand,et al.  Exploring the relationships between design measures and software quality in object-oriented systems , 2000, J. Syst. Softw..

[17]  Edward F. Weller.  Using metrics to manage software projects , 1994, Computer.

[18]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[19]  Daniel Port,et al.  Comparing Model Generated with Expert Generated IV&V Activity Plans , 2007, ESEM 2007.

[20]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[21]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[22]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[23]  Frank Bomarius,et al.  COBRA: a hybrid method for software cost estimation, benchmarking, and risk assessment , 1998, Proceedings of the 20th International Conference on Software Engineering.

[24]  Claes Wohlin,et al.  Capture-recapture in software inspections after 10 years research--theory, evaluation and application , 2004, J. Syst. Softw..

[25]  Barry W. Boehm,et al.  What we have learned about fighting defects , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[26]  Rob J. Kusters,et al.  Identification of factors that influence defect injection and detection in development of software intensive products , 2007, Inf. Softw. Technol..

[27]  Capers Jones,et al.  Applied software measurement (2nd ed.): assuring productivity and quality , 1996 .

[28]  Stephen H. Kan,et al.  Metrics and Models in Software Quality Engineering , 1994, SOEN.

[29]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[30]  Claes Wohlin,et al.  Defect content estimations from review data , 1998, Proceedings of the 20th International Conference on Software Engineering.

[31]  Daniel Port,et al.  Comparing Model Generated with Expert Generated IV&V Activity Plans , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[32]  Lionel C. Briand,et al.  Quantitative evaluation of capture-recapture models to control software inspections , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[33]  Stephen G. Eick,et al.  Estimating software fault content before coding , 1992, International Conference on Software Engineering.

[34]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[35]  N. Kohtake,et al.  Software Independent Verification and Validation for Spacecraft at JAXA , 2008, 2008 IEEE Aerospace Conference.

[36]  Lionel C. Briand,et al.  Using multiple adaptive regression splines to support decision making in code inspections , 2004, J. Syst. Softw..

[37]  Capers Jones,et al.  Applied software measurement: assuring productivity and quality , 1991 .

[38]  Jürgen Münch,et al.  The Use of Simulation Techniques for Hybrid Software Cost Estimation and Risk Analysis , 2008, Adv. Comput..

[39]  Claes Wohlin,et al.  State‐of‐the‐art: software inspections after 25 years , 2002, Softw. Test. Verification Reliab..

[40]  M. Kendall,et al.  The Problem of $m$ Rankings , 1939 .

[41]  Ronald L. Wasserstein,et al.  Monte Carlo: Concepts, Algorithms, and Applications , 1997 .

[42]  H. E. Dunsmore,et al.  Software engineering metrics and models , 1986 .

[43]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[44]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[45]  Ieee Std,et al.  IEEE Standard for Software Verification and Validation , 2008 .

[46]  David Vose,et al.  Quantitative Risk Analysis: A Guide to Monte Carlo Simulation Modelling , 1996 .

[47]  Ioannis Stamelos,et al.  Software Defect Prediction Using Regression via Classification , 2006, IEEE International Conference on Computer Systems and Applications, 2006..

[48]  Barry Boehm,et al.  Determining how much software assurance is enough? A value-based approach , 2005, ISESE.

[49]  Lionel C. Briand,et al.  A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content , 2000, IEEE Trans. Software Eng..