Test case selection and prioritization: risk-based or design-based?

The objective of this qualitative study was to observe and empirically study how software organizations decide on which test cases to select for their software projects. As the software test processes are limited in resources such as time or money, a selection process usually exists for tested features. In this study we conducted a survey on 31 software-producing organizations, and interviewed 36 software professionals from 12 focus organizations to gain a better insight into testing practices. Our findings indicated that the basic approaches to test case selection are usually oriented towards two possible objectives. One is the risk-based selection, where the aim is to focus testing on those parts that are too expensive to fix after launch. The other is design-based selection, where the focus is on ensuring that the software is capable of completing the core operations it was designed to do. These results can then be used to develop testing organizations and to identify better practices for test case selection.

[1]  James A. Whittaker,et al.  What is software testing? And why is it so hard? , 2000 .

[2]  Béatrix Barafort,et al.  Benefits Resulting from the Combined Use of ISO/IEC 15504 with the Information Technology Infrastructure Library (ITIL) , 2002, PROFES.

[3]  Dirk Muthig,et al.  Making testing product lines more efficient by improving the testability of product line architectures , 2006, ROSATEA '06.

[4]  Michael D. Myers,et al.  A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems , 1999, MIS Q..

[5]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[6]  Ossi Taipale,et al.  Improving software testing by observing practice , 2006, ISESE '06.

[7]  Gregory Tassey,et al.  Prepared for what , 2007 .

[8]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[9]  Felix Redmill,et al.  Exploring risk‐based testing and its implications , 2004, Softw. Test. Verification Reliab..

[10]  Pamela Jordan.  Basics of qualitative research: Grounded theory procedures and techniques , 1994 .

[11]  Nasib S. Gill.  Factors affecting effective software quality management revisited , 2005, SOEN.

[12]  Robert J. Vokurka,et al.  New product introduction delays in the computer industry , 2000, Ind. Manag. Data Syst..

[13]  Edward Kit,et al.  Software testing in the real world - improving the process , 1995 .

[14]  Katja Karhu,et al.  Empirical Observations on Software Testing Automation , 2009, 2009 International Conference on Software Testing Verification and Validation.

[15]  Cem Kaner,et al.  Testing Computer Software, Second Edition , 1993 .

[16]  Katja Karhu,et al.  Outsourcing and Knowledge Management in Software Testing , 2007, EASE.

[17]  J. Olsen,et al.  The European Commission , 2020, The European Union.

[18]  Yanping Chen,et al.  Specification-based regression test selection with risk analysis , 2002, CASCON.

[19]  Bertrand Meyer.  Design and code reviews in the age of the internet , 2008, CACM.

[20]  H. Hulkko,et al.  A multiple case study on the impact of pair programming on product quality , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[21]  K. Eisenhardt.  Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[22]  Nathan H. Petschenik.  Practical Priorities in System Testing , 1985, IEEE Software.

[23]  Gregg Rothermel,et al.  An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models , 2006, SIGSOFT '06/FSE-14.

[24]  Barry W. Boehm,et al.  How Much Software Quality Investment Is Enough: A Value-Based Approach , 2006, IEEE Software.

[25]  Gregg Rothermel,et al.  On test suite composition and cost-effective regression testing , 2004, TSEM.

[26]  N. Leech,et al.  Validity and Qualitative Research: An Oxymoron? , 2007 .

[27]  Barry W. Boehm,et al.  Get Ready for Agile Methods, with Care , 2002, Computer.

[28]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[29]  Guy Paré,et al.  Using case study research to build theories of IT implementation , 1997 .

[30]  Chengying Mao,et al.  Regression testing for component-based software via built-in test design , 2007, SAC '07.

[31]  Ossi Taipale,et al.  Analysis of Problems in Testing Practices , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[32]  Kasper Hornbæk,et al.  What do usability evaluators do in practice?: an explorative study of think-aloud testing , 2006, DIS '06.

[33]  Bruce Elliott,et al.  Testing Computer Software , 1994 .

[34]  Antonia Bertolino.  The (Im)maturity level of software testing , 2004, SOEN.

[35]  Qingxin Shi,et al.  A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests , 2008, NordiCHI.

[36]  Jeffrey M. Voas,et al.  Software testability: an experiment in measuring simulation reusability , 1995, SSR '95.