Metrics in Software Test Planning and Test Design Processes

Software metrics plays an important role in measuring attributes that are critical to the success of a software project. Measurement of these attributes helps to make the characteristics and relationships between the attributes clearer. This in turn supports informed decision making. The field of software engineering is affected by infrequent, incomplete and inconsistent measurements. Software testing is an integral part of software development, providing opportunities for measurement of process attributes. The measurement of software testing process attributes enables the management to have better insight in to the software testing process. The aim of this thesis is to investigate the metric support for software test planning and test design processes. The study comprises of an extensive literature study and follows a methodical approach. This approach consists of two steps. The first step comprises of analyzing key phases in software testing life cycle, inputs required for starting the software test planning and design processes and metrics indicating the end of software test planning and test design processes. After establishing a basic understanding of the related concepts, the second step identifies the attributes of software test planning and test design processes including metric support for each of the identified attributes. The results of the literature survey showed that there are a number of different measurable attributes for software test planning and test design processes. The study partitioned these attributes in multiple categories for software test planning and test design processes. For each of these attributes, different existing measurements are studied. A consolidation of these measurements is presented in this thesis which is intended to provide an opportunity for management to consider improvement in these processes.

[1]  Paul Ammann,et al.  Using model checking to generate tests from specifications , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[2]  Norman E. Fenton,et al.  Quantitative Analysis of Faults and Failures in a Complex Software System , 2000, IEEE Trans. Software Eng..

[3]  Dieter Kranzlmuller,et al.  Event Graph Analysis for Debugging Massively Parallel Programs , 2000 .

[4]  Elizabeth K. Bailey,et al.  Software Measurement for DoD Systems: Recommendations for Initial Core Measures. , 1992 .

[5]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[6]  Cem Kaner,et al.  Software Engineering Metrics: What Do They Measure and How Do We Know? , 2004 .

[7]  William A. Florac,et al.  Goal-Driven Software Measurement. A Guidebook. , 1996 .

[8]  Charles W. Butler,et al.  Design complexity measurement and testing , 1989, CACM.

[9]  Rex Black Managing the Testing Process , 1999 .

[10]  Daniel J. Paulish,et al.  Software metrics - a practitioner's guide to improved product development , 1993, Chapman & Hall computing series.

[11]  Brian Marick New Models for Test Development , 1999 .

[12]  Harvey P. Siy,et al.  Predicting Fault Incidence Using Software Change History , 2000, IEEE Trans. Software Eng..

[13]  Edsger W. Dijkstra,et al.  Structured programming , 1972, A.P.I.C. Studies in data processing.

[14]  Kjetil Molkken,et al.  A Review of Surveys on Software Effort Estimation , 2003 .

[15]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[16]  Shanker Sanyal,et al.  Defect tracking and reliability modeling for a new product release , 1992, CASCON.

[17]  Steve McConnell,et al.  Daily Build and Smoke Test , 1996, IEEE Software.

[18]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[19]  C. Robert Carlson,et al.  Developing a Testing Maturity Model for software test process evaluation and improvement , 1996, Proceedings International Test Conference 1996. Test and Design Validity.

[20]  Sandro Morasca,et al.  Towards a theoretical framework for measuring software attributes , 1997, Proceedings Fourth International Software Metrics Symposium.

[21]  Jim Gray,et al.  Why Do Computers Stop and What Can Be Done About It? , 1986, Symposium on Reliability in Distributed Software and Database Systems.

[22]  Robert B. Grady,et al.  Practical Software Metrics for Project Management and Process Improvement , 1992 .

[23]  L. Lauterbach,et al.  Experimental evaluation of six test techniques , 1989, Proceedings of the Fourth Annual Conference on Computer Assurance, 'Systems Integrity, Software Safety and Process Security.

[24]  Ilene Burnstein,et al.  Practical Software Testing , 2003, Springer Professional Computing.

[25]  Keijo Heljanko,et al.  Specification coverage aided test selection , 2003, Third International Conference on Application of Concurrency to System Design, 2003. Proceedings..

[26]  Cem Kaner,et al.  Measuring the Effectiveness of Software Testers , 2003 .

[27]  Stefan P. Jaskiel,et al.  Systematic Software Testing , 2002 .

[28]  W. R. Howard Software Measurement and Estimation: A Practical Approach , 2007 .

[29]  Ram Chillarege Software Testing Best Practices , 1999 .

[30]  Paul Piwowarski,et al.  Coverage measurement experience during function test , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[31]  Robert Fantina Practical Software Process Improvement , 2005 .

[32]  Victor R. Basili,et al.  Comparing the Effectiveness of Software Testing Strategies , 1987, IEEE Transactions on Software Engineering.

[33]  Niclas Ohlsson,et al.  Predicting Fault-Prone Software Modules in Telephone Switches , 1996, IEEE Trans. Software Eng..

[34]  Paul Ammann,et al.  A specification-based coverage metric to evaluate test sets , 1999, Proceedings 4th IEEE International Symposium on High-Assurance Systems Engineering.

[35]  Bogdan Korel,et al.  Black-box test reduction using input-output analysis , 2000, ISSTA '00.

[36]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[37]  Gregg Rothermel,et al.  An empirical study of regression test selection techniques , 2001, ACM Trans. Softw. Eng. Methodol..

[38]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[39]  Elaine J. Weyuker,et al.  Predicting the location and number of faults in large software systems , 2005, IEEE Transactions on Software Engineering.

[40]  William Lewis,et al.  Software Testing and Continuous Quality Improvement , 2000 .

[41]  J. P. Lewis Limits to software estimation , 2001, SOEN.

[42]  John A. Clark,et al.  INVESTIGATING THE APPLICABILITY OF TRADITIONAL TEST ADEQUACY CRITERIA FOR OBJECT-ORIENTED PROGRAMS , 2000 .

[43]  Jeff Tian,et al.  Software quality engineering - testing, quality assurance, and quantifiable improvement , 2005 .

[44]  William E. Lewis,et al.  Software Testing and Continuous Quality Improvement, Second Edition , 2004 .

[45]  Kjetil Moløkken-Østvold,et al.  A review of software surveys on software effort estimation , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[46]  Antonia Bertolino,et al.  Using Spanning Sets for Coverage Testing , 2003, IEEE Trans. Software Eng..

[47]  Kishor S. Trivedi,et al.  Extended Classification of Software Faults Based on Aging , 2001 .

[48]  M. Megel,et al.  Research Design , 2019, Beiträge zur Landschafts- und Umweltplanung I Contributions to Landscape and Environmental Planning.

[49]  Anita D. Carleton,et al.  Case studies of software-process-improvement measurement , 1994, Computer.

[50]  Mats Per Erik Heimdahl,et al.  Specification test coverage adequacy criteria = specification test generation inadequacy criteria , 2004, Eighth IEEE International Symposium on High Assurance Systems Engineering, 2004. Proceedings..

[51]  Stephen H. Kan,et al.  Metrics and Models in Software Quality Engineering , 1994, SOEN.

[52]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[53]  Bojan Cukic,et al.  Robust prediction of fault-proneness by random forests , 2004, 15th International Symposium on Software Reliability Engineering.

[54]  Robert M. Poston,et al.  Evaluating and selecting testing tools , 1992, IEEE Software.

[55]  Elfriede Dustin,et al.  Effective software testing : 50 specific ways to improve your testing , 2002 .

[56]  Jooyoung Seo,et al.  Tailoring test process by using the component-based development paradigm and the XML technology , 2000, Proceedings Seventh Asia-Pacific Software Engeering Conference. APSEC 2000.

[57]  Gregg Rothermel,et al.  Modeling the cost-benefits tradeoffs for regression testing techniques , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[58]  N. Falconer Structured Programming , 1973, Nature.

[59]  C. C. Trillingham The Philosophy of Testing , 1933 .

[60]  Antony Black Critical Testing Process: Plan, Prepare, Perform, Perfect , 2003 .

[61]  Steven R. Rakitin,et al.  Software verification and validation for practitioners and managers , 2001 .

[62]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[63]  Ajitha Rajan,et al.  Coverage metrics for requirements-based testing , 2006, ISSTA '06.

[64]  R. Spital Automated Software Testing , 1983 .

[65]  Taghi M. Khoshgoftaar,et al.  The Detection of Fault-Prone Programs , 1992, IEEE Trans. Software Eng..

[66]  Taghi M. Khoshgoftaar,et al.  Early Quality Prediction: A Case Study in Telecommunications , 1996, IEEE Softw..

[67]  David S. Rosenblum,et al.  Predicting the cost-effectiveness of regression testing strategies , 1996, SIGSOFT '96.

[68]  Yuri Chernak,et al.  Validating and Improving Test-Case Effectiveness , 2001, IEEE Softw..

[69]  Paul W. Oman,et al.  Using metrics to manage the end-game of a software project , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[70]  W. H. Flank A Philosophy for Testing , 1989 .

[71]  Marnie L. Hutcheson,et al.  Software testing fundamentals - methods and metrics , 2003 .

[72]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[73]  Rakesh Agarwal,et al.  Software estimation: a fuzzy approach , 2006, SOEN.