Software Testing Process Models Benefits & Drawbacks: a Systematic Literature Review

Context: Software testing plays an essential role in product quality improvement. For this reason, several software testing models have been developed to support organizations. However, adoption of testing process models inside organizations is still sporadic, with a need for more evidence about reported experiences. Aim: Our goal is to identify results gathered from the application of software testing models in organizational contexts. We focus on characteristics such as the context of use, practices applied in different testing process phases, and reported benefits & drawbacks. Method: We performed a Systematic Literature Review (SLR) focused on studies about the application of software testing processes, complemented by results from previous reviews. Results: From 35 primary studies and survey-based articles, we collected 17 testing models. Although most of the existing models are described as applicable to general contexts, the evidence obtained from the studies shows that some models are not suitable for all enterprise sizes, and inadequate for specific domains. Conclusion: The SLR evidence can serve to compare different software testing models for applicability inside organizations. Both benefits and drawbacks, as reported in the surveyed cases, allow getting a better view of the strengths and weaknesses of each model.

[1]  Stuart Christopher Reid.  BS 7925-2: the software component testing standard , 2000, Proceedings First Asia-Pacific Conference on Quality Software.

[2]  Bruno Rossi,et al.  Is Mutation Testing Ready to Be Adopted Industry-Wide? , 2016, PROFES.

[3]  D. Rennie.  Grounded Theory Methodology , 1998 .

[4]  Tore Dybå,et al.  Factors of software process improvement success in small and large organizations: an empirical study in the scandinavian context , 2003, ESEC/FSE-11.

[5]  Michael Steiner,et al.  Make test process assessment similar to software process assessment—the Test SPICE approach , 2012, J. Softw. Evol. Process..

[6]  Werasak Kurutach,et al.  An analysis of software process improvement for sustainable development in Thailand , 2008, 2008 8th IEEE International Conference on Computer and Information Technology.

[7]  Jos J. M. Trienekens,et al.  Towards a Metrics Based Verification and Validation Maturity Model , 2002, SAFECOMP.

[8]  Vahid Garousi,et al.  Software test maturity assessment and test process improvement: A multivocal literature review , 2017, Inf. Softw. Technol..

[9]  Claes Wohlin,et al.  Context in industrial software engineering research , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[10]  Anita D. Carleton,et al.  Case studies of software-process-improvement measurement , 1994, Computer.

[11]  Per Runeson,et al.  A minimal test practice framework for emerging software organizations , 2005, Softw. Test. Verification Reliab..

[12]  Tihana Galinac,et al.  Empirical evaluation of selected best practices in implementation of software process improvement , 2009 .

[13]  Pankaj Kamthan,et al.  Managing Corrective Actions to Closure in Open Source Software Test Process , 2013, SEKE.

[14]  K. A. Hayden,et al.  State-of-the-evidence reviews: advantages and challenges of including grey literature. , 2006, Worldviews on evidence-based nursing.

[15]  Yang Sun,et al.  The research of testing process and data tracing based on hybrid model , 2015, 2015 4th International Conference on Computer Science and Network Technology (ICCSNT).

[16]  Jongmoon Baik,et al.  A Strategic Test Process Improvement Approach Using an Ontological Description for MND-TMM , 2008, Seventh IEEE/ACIS International Conference on Computer and Information Science (icis 2008).

[17]  Standard Glossary of Software Engineering Terminology , 1990 .

[18]  R. Nickerson.  Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[19]  Adriano Bessa Albuquerque,et al.  The Definiton of a Testing Process to Small-Sized Companies: The Brazilian Scenario , 2010, 2010 Seventh International Conference on the Quality of Information and Communications Technology.

[20]  Lea Hannola,et al.  Extending Process Improvement into a New Organizational Unit , 2009, 2009 Australian Software Engineering Conference.

[21]  Leon Sterling,et al.  Towards the software engineering of neural networks: a maturity model , 2000, Proceedings 2000 Australian Software Engineering Conference.

[22]  C. Robert Carlson,et al.  Developing a Testing Maturity Model for software test process evaluation and improvement , 1996, Proceedings International Test Conference 1996. Test and Design Validity.

[23]  Kai Petersen,et al.  Analyzing an automotive testing process with evidence-based software engineering , 2013, Inf. Softw. Technol..

[24]  Li Xu-Xiang,et al.  The PDCA-based software testing improvement framework , 2010, The 2010 International Conference on Apperceiving Computing and Intelligence Analysis Proceeding.

[25]  Barbara A. Kitchenham,et al.  Would wider adoption of reproducible research be beneficial for empirical software engineering research? , 2017, J. Intell. Fuzzy Syst..

[26]  Reiner R. Dumke,et al.  Evaluation Approaches in Software Testing , 2008 .

[27]  D. Ross Jeffery,et al.  An exploratory study of process enactment as input to software process improvement , 2006, WoSQ '06.

[28]  Fabiano Cutigi Ferrari,et al.  Characterising the state of the practice in software testing through a TMMi-based process , 2015, Journal of Software Engineering Research and Development.

[29]  Wasif Afzal,et al.  Software test process improvement approaches: A systematic literature review and an industrial case study , 2016, J. Syst. Softw..

[30]  Bruno Rossi,et al.  Requirements prioritization in software engineering: A systematic mapping study , 2013, 2013 3rd International Workshop on Empirical Requirements Engineering (EmpiRE).

[31]  Ossi Taipale,et al.  Improving software testing by observing practice , 2006, ISESE '06.

[32]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[33]  M. Scriven.  Evaluation thesaurus, 4th ed. , 1991 .

[34]  Vahid Garousi,et al.  A survey of software testing practices in Canada , 2013, J. Syst. Softw..

[35]  Daniela Cruzes,et al.  What works for whom, where, when, and why? On the role of context in empirical software engineering , 2012, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.

[36]  A. Farooq,et al.  A meta-measurement approach for software test processes , 2008, 2008 IEEE International Multitopic Conference.

[37]  Kai Petersen,et al.  Worldviews, Research Methods, and their Relationship to Validity in Empirical Software Engineering Research , 2013, 2013 Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement.

[38]  Raimundas Matulevicius,et al.  Empirical Analysis of the Test Maturity Model Integration (TMMi) , 2013, ICIST.

[39]  Hossein Saiedian,et al.  Characterizing a software process maturity model for small organizations , 1997, SICE.

[40]  M. Nobre,et al.  The PICO strategy for the research question construction and evidence search. , 2007, Revista latino-americana de enfermagem.

[41]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[42]  Ivan A. Garcia,et al.  Adopting an RIA-Based Tool for Supporting Assessment, Implementation and Learning in Software Process Improvement under the NMX-I-059/02-NYCE-2005 Standard in Small Software Enterprises , 2010, 2010 Eighth ACIS International Conference on Software Engineering Research, Management and Applications.

[43]  Santiago Matalonga,et al.  Matching Context Aware Software Testing Design Techniques to ISO/IEC/IEEE 29119 , 2015, SPICE.

[44]  Daniela Cruzes,et al.  Synthesizing evidence in software engineering research , 2010, ESEM '10.

[45]  Sigrid Eldh,et al.  Towards a Test Automation Improvement Model (TAIM) , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[46]  Chongwon Lee,et al.  Adapting and adjusting test process reflecting characteristics of embedded software and industrial properties based on referential models , 2009, ICIS.

[47]  Tore Dybå,et al.  Evidence-based software engineering , 2016, Perspectives on Data Science for Software Engineering.

[48]  Jean-Martin Simon.  SPICE: Overview for software process improvement , 1996, J. Syst. Archit..

[49]  Per Runeson,et al.  A Self-assessment Framework for Finding Improvement Objectives with ISO/IEC 29119 Test Standard , 2011, EuroSPI.

[50]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[51]  Mika Katara,et al.  A Test Process Improvement Model for Automated Test Generation , 2012, PROFES.

[52]  A. Strauss,et al.  Grounded theory methodology: An overview. , 1994 .

[53]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[54]  Mikko Nieminen,et al.  Adaptable Design for Root Cause Analysis of a Model-Based Software Testing Process , 2015, 2015 12th International Conference on Information Technology - New Generations.

[55]  Vahid Garousi,et al.  The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature , 2016, EASE.

[56]  Eun Jung.  A Test Process Improvement Model for Embedded Software Developments , 2009, 2009 Ninth International Conference on Quality Software.

[57]  Munoz Mirna,et al.  Advantages of Using a Multi-model Environment in Software Process Improvement , 2011, 2011 IEEE Electronics, Robotics and Automotive Mechanics Conference.

[58]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[59]  Kai Petersen,et al.  Guidelines for conducting systematic mapping studies in software engineering: An update , 2015, Inf. Softw. Technol..

[60]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[61]  Ossi Taipale,et al.  How Test Organizations Adopt New Testing Practices and Methods? , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[62]  Byoungju Choi,et al.  Optimizing Test Process Action Plans by Blending Testing Maturity Model and Design of Experiments , 2008, 2008 The Eighth International Conference on Quality Software.

[63]  Marcelo Schneck de Paula Pessôa,et al.  Test Process Models: Systematic Literature Review , 2014, SPICE.

[64]  R. Baskaran,et al.  Experimental design on defect analysis in software process improvement , 2012, 2012 International Conference on Recent Advances in Computing and Software Systems.

[65]  Mauro Pezzè,et al.  Software testing and analysis - process, principles and techniques , 2007 .

[66]  Shaukat Ali,et al.  Formalizing the ISO/IEC/IEEE 29119 Software Testing Standard , 2015, 2015 ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS).

[67]  Ayaz Farooq,et al.  An evaluation framework for software test processes , 2009 .

[68]  Dirk Eisenbiegler,et al.  Schwarzwälder — An online test framework , 2012, 2012 6th IEEE International Conference on Digital Ecosystems and Technologies (DEST).

[69]  Ermeson Carneiro de Andrade,et al.  MPT.BR: A Brazilian Maturity Model for Testing , 2012, 2012 12th International Conference on Quality Software.

[70]  Tanja Toroi,et al.  Identifying Process Improvement Targets in Test Processes: A Case Study , 2013, 2013 IEEE International Conference on Software Maintenance.

[71]  Giancarlo Succi,et al.  Modelling Failures Occurrences of Open Source Software with Reliability Growth , 2010, OSS.

[72]  Elfriede Dustin,et al.  Effective software testing : 50 specific ways to improve your testing , 2002 .

[73]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[74]  Fergal McCaffery,et al.  Tailoring software process capability/maturity models for the health domain , 2013 .

[75]  Fabiano Cutigi Ferrari,et al.  Identifying a Subset of TMMi Practices to Establish a Streamlined Software Testing Process , 2013, SBES.