Hybrid Is Better: Why and How Test Coverage and Software Reliability Can Benefit Each Other

Functional, structural and operational testing are three broad categories of software testing methods driven by the product functionalities, the way it is implemented, and the way it is expected to be used, respectively. A large body of the software testing literature is devoted to evaluate and compare test techniques in these categories. Although it appears reasonable to devise hybrid methods to merge their different strengths - because different techniques may complement each other by targeting different types of faults and/or using different artifacts - we still miss clear guidelines on how to best combine them.

[1]  Kim Herzig.  There's never enough time to do all the testing you want , 2016, Perspectives on Data Science for Software Engineering.

[2]  Dalal Alrmuny.  A Comparative Study of Test Coverage-Based Software Reliability Growth Models , 2014, 2014 11th International Conference on Information Technology: New Generations.

[3]  Fabio Del Frate,et al.  On the correlation between code coverage and software reliability , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[4]  Domenico Cotroneo,et al.  Combining Operational and Debug Testing for Improving Reliability , 2013, IEEE Transactions on Reliability.

[5]  Marco Kuhrmann,et al.  Is 100% Test Coverage a Reasonable Requirement? Lessons Learned from a Space Software Project , 2017, PROFES.

[6]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[7]  Domenico Cotroneo,et al.  A learning-based method for combining testing techniques , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[8]  Marc Roper.  Software testing - searching for the missing link , 1999, Inf. Softw. Technol..

[9]  Martin Neil,et al.  Building large-scale Bayesian networks , 2000, The Knowledge Engineering Review.

[10]  David Notkin,et al.  Checking inside the black box: regression testing by comparing value spectra , 2005, IEEE Transactions on Software Engineering.

[11]  Brian Marick.  How to Misuse Code Coverage , 1999 .

[12]  Peng Lu,et al.  Test-Execution-Based Reliability Measurement and Modeling for Large Commercial Software , 1995, IEEE Trans. Software Eng..

[13]  Michael R. Lyu.  Software Reliability Engineering: A Roadmap , 2007, Future of Software Engineering (FOSE '07).

[14]  Bev Littlewood,et al.  Modelling the Effects of Combining Diverse Software Fault Detection Techniques , 2008, Formal Methods and Testing.

[15]  John D. Musa,et al.  A theory of software reliability and its application , 1975, IEEE Transactions on Software Engineering.

[16]  Bojan Cukic,et al.  Software reliability corroboration , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[17]  John D. Musa,et al.  Operational profiles in software-reliability engineering , 1993, IEEE Software.

[18]  Gregg Rothermel,et al.  An empirical investigation of program spectra , 1998, PASTE '98.

[19]  Tsong Yueh Chen,et al.  Adaptive Random Testing , 2004, ASIAN.

[20]  Rui Abreu,et al.  A Survey on Software Fault Localization , 2016, IEEE Transactions on Software Engineering.

[21]  Kai-Yuan Cai,et al.  Optimal and adaptive testing for software reliability assessment , 2004, Inf. Softw. Technol..

[22]  Antonia Bertolino,et al.  An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing , 2017, Software Quality Journal.

[23]  Domenico Cotroneo,et al.  RELAI Testing: A Technique to Assess and Improve Software Reliability , 2016, IEEE Transactions on Software Engineering.

[24]  Ron Patton,et al.  Software Testing , 2000 .

[25]  Phyllis G. Frankl,et al.  Comparison of delivered reliability of branch, data flow and operational testing: A case study , 2000, ISSTA '00.

[26]  John D. Musa,et al.  Best current practice of SRE , 1996 .

[27]  Bertrand Meyer,et al.  Is Branch Coverage a Good Measure of Testing Effectiveness? , 2010, LASER Summer School.

[28]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..

[29]  Stefano Russo,et al.  Adaptive Coverage and Operational Profile-Based Testing for Reliability Improvement , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[30]  Stefano Russo,et al.  Probabilistic Sampling-Based Testing for Accelerated Reliability Assessment , 2018, 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[31]  Carol S. Smidts,et al.  Software testing with an operational profile: OP definition , 2014, CSUR.

[32]  Gregory Gay,et al.  The Risks of Coverage-Directed Test Case Generation , 2015, IEEE Transactions on Software Engineering.

[33]  Bojan Cukic,et al.  A Bayesian approach to reliability prediction and assessment of component based systems , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[34]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[35]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[36]  Stefano Russo,et al.  On Adaptive Sampling-Based Testing for Software Reliability Assessment , 2016, 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE).

[37]  Fouad ben Nasr Omri.  Weighted statistical white-box testing with proportional-optimal stratification , 2014, WCOP '14.

[38]  Bev Littlewood,et al.  Evaluating Testing Methods by Delivered Reliability , 1998, IEEE Trans. Software Eng..

[39]  Edward N. Adams,et al.  Optimizing Preventive Service of Software Products , 1984, IBM J. Res. Dev..

[40]  Aditya P. Mathur,et al.  Software testing and reliability , 1996 .

[41]  Antonia Bertolino,et al.  Does Code Coverage Provide a Good Stopping Rule for Operational Profile Based Testing? , 2016, 2016 IEEE/ACM 11th International Workshop in Automation of Software Test (AST).

[42]  Akbar Siami Namin,et al.  Prioritizing Mutation Operators Based on Importance Sampling , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[43]  Edsger W. Dijkstra,et al.  Structured programming , 1972, A.P.I.C. Studies in data processing.

[44]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[45]  RothermelGregg,et al.  An empirical investigation of program spectra , 1998 .