Testing techniques selection based on ODC fault types and software metrics

Software testing techniques differ in the type of faults they are more prone to detect, and their performance varies depending on the features of the application being tested. Practitioners often use informally their knowledge about the software under test in order to combine testing techniques for maximizing the number of detected faults. This work presents an approach to enable practitioners to select testing techniques according to the features of the software to test. A method to build a testing-related base of knowledge for tailoring the techniques selection process to the specific application(s) is proposed. The method grounds upon two basic steps: (i) constructing, on an empirical basis, models to characterize the software to test in terms of fault types it is more prone to contain; (ii) characterizing testing techniques with respect to fault types they are more prone to detect in the given context. Using the created base of knowledge, engineers within an organization can define the mix of techniques so as to maximize the effectiveness of the testing process for their specific software.

[1]  Sira Vegas,et al.  A Characterisation Schema for Software Testing Techniques , 2005, Empirical Software Engineering.

[2]  Richard G. Hamlet,et al.  Partition Testing Does Not Inspire Confidence , 1990, IEEE Trans. Software Eng..

[3]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[4]  Ramanath Subramanyam,et al.  Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects , 2003, IEEE Trans. Software Eng..

[5]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[6]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[7]  Siddhartha R. Dalal,et al.  Using Defect Patterns to Uncover Opportunities for Improvement , 1999 .

[8]  Bev Littlewood,et al.  Modeling the Effects of Combining Diverse Software Fault Detection Techniques , 2000, IEEE Trans. Software Eng..

[9]  Swapna S. Gokhale,et al.  Regression Tree Modeling For The Prediction Of Software Quality , 1997 .

[10]  Kai-Yuan Cai,et al.  Adaptive software testing with fixed-memory feedback , 2007, J. Syst. Softw..

[11]  Hélène Waeselynck,et al.  An investigation of statistical software testing , 1991, Softw. Test. Verification Reliab..

[12]  Kishor S. Trivedi,et al.  Software Reliability and Testing Time Allocation: An Architecture-Based Approach , 2010, IEEE Transactions on Software Engineering.

[13]  John S. Gourlay A Mathematical Framework for the Investigation of Testing , 1983, IEEE Transactions on Software Engineering.

[14]  Jeff Tian,et al.  Measuring and Modeling Usage and Reliability for Statistical Web Testing , 2001, IEEE Trans. Software Eng..

[15]  Daniel P. Siewiorek,et al.  Automated robustness testing of off-the-shelf software components , 1998, Digest of Papers. Twenty-Eighth Annual International Symposium on Fault-Tolerant Computing (Cat. No.98CB36224).

[16]  R. R. Hocking The analysis and selection of variables in linear regression , 1976 .

[17]  Yves Crouzet,et al.  Benchmarking the dependability of Windows and Linux using PostMark/spl trade/ workloads , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[18]  Stephen R. Schach,et al.  Validation of the coupling dependency metric as a predictor of run-time failures and maintenance measures , 1998, Proceedings of the 20th International Conference on Software Engineering.

[19]  Victor R. Basili,et al.  Comparing the Effectiveness of Software Testing Strategies , 1987, IEEE Transactions on Software Engineering.

[20]  Roberto Pietrantuono,et al.  Reliability-Oriented Verification of Mission-Critical Software Systems , 2009 .

[21]  Bev Littlewood,et al.  Evaluating Testing Methods by Delivered Reliability , 1998, IEEE Trans. Software Eng..

[22]  James Miller,et al.  Comparing and combining software defect detection techniques: a replicated empirical study , 1997, ESEC '97/FSE-5.

[23]  Mauro Pezzè,et al.  Software testing and analysis - process, principles and techniques , 2007 .

[24]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[25]  James A. Whittaker,et al.  A Markov Chain Model for Statistical Software Testing , 1994, IEEE Trans. Software Eng..

[26]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[27]  Henrique Madeira,et al.  Emulation of Software Faults: A Field Data Study and a Practical Approach , 2006, IEEE Transactions on Software Engineering.

[28]  C. Trammell,et al.  Quantifying the reliability of software: statistical testing based on a usage model , 1995, Proceedings of Software Engineering Standards Symposium.

[29]  John A. Clark,et al.  Efficient Software Verification: Statistical Testing Using Automated Search , 2010, IEEE Transactions on Software Engineering.

[30]  Elaine J. Weyuker,et al.  Comparison of program testing strategies , 1991, TAV4.

[31]  Kishor S. Trivedi,et al.  Analysis of Software Aging in a Web Server , 2006, IEEE Transactions on Reliability.

[32]  Martin Hiller,et al.  An experimental comparison of fault and error injection , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[33]  Chryssa Dislis,et al.  Test process optimization: closing the gap in the defect spectrum , 1999, International Test Conference 1999. Proceedings (IEEE Cat. No.99CH37034).

[34]  Elaine J. Weyuker,et al.  A Formal Analysis of the Fault-Detecting Ability of Testing Methods , 1993, IEEE Trans. Software Eng..

[35]  Anup K. Ghosh,et al.  Testing the robustness of Windows NT software , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[36]  Elaine J. Weyuker,et al.  Deriving workloads for performance testing , 1996 .

[37]  Henrique Madeira,et al.  Xception: Software Fault Injection and Monitoring in Processor Functional Units1 , 1995 .

[38]  Kerong Ben,et al.  Software Metrics Reduction for Fault-Proneness Prediction of Software Modules , 2010, NPC.

[39]  Bev Littlewood,et al.  Modelling the Effects of Combining Diverse Software Fault Detection Techniques , 2008, Formal Methods and Testing.

[40]  Victor R. Basili,et al.  A Validation of Object-Oriented Design Metrics as Quality Indicators , 1996, IEEE Trans. Software Eng..

[41]  Ilka Philippow,et al.  UML-Based Statistical Test Case Generation , 2002, NetObjectDays.

[42]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[43]  Stefan Wagner Software Quality Economics for Combining Defect-Detection Techniques⋆ , 2005 .

[44]  Padmanabhan Santhanam,et al.  Evaluating the software test strategy for the 2000 Sydney Olympics , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[45]  Yves Crouzet,et al.  An experimental study on software structural testing: deterministic versus random input generation , 1991, [1991] Digest of Papers. Fault-Tolerant Computing: The Twenty-First International Symposium.

[46]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..

[47]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[48]  Elaine J. Weyuker,et al.  Predicting the location and number of faults in large software systems , 2005, IEEE Transactions on Software Engineering.

[49]  Niclas Ohlsson,et al.  Predicting Fault-Prone Software Modules in Telephone Switches , 1996, IEEE Trans. Software Eng..

[50]  Ram Chillarege,et al.  Generation of an error set that emulates software faults based on field data , 1996, Proceedings of Annual Symposium on Fault Tolerant Computing.

[51]  A. Jefferson Offutt,et al.  An evaluation of combination strategies for test case selection , 2006, Empirical Software Engineering.

[52]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[53]  William E. Lewis,et al.  Software Testing and Continuous Quality Improvement, Second Edition , 2004 .

[54]  Elaine J. Weyuker Comparing the Effectiveness of Testing Techniques , 2008, Formal Methods and Testing.

[55]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[56]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[57]  Pascale Thévenod-Fosse Software Validation by Means of Statistical Testing: Retrospect and Future Direction , 1991 .

[58]  Banu Diri,et al.  A systematic review of software fault prediction studies , 2009, Expert Syst. Appl..

[59]  Jean Arlat,et al.  Dependability of COTS Microkernel-Based Systems , 2002, IEEE Trans. Computers.

[60]  Dawson R. Engler,et al.  Some Lessons from Using Static Analysis and Software Model Checking for Bug Finding , 2003, SoftMC@CAV.

[61]  Domenico Cotroneo,et al.  Workload Characterization for Software Aging Analysis , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[62]  Richard G. Hamlet,et al.  Theoretical comparison of testing methods , 1989, TAV3.

[63]  S. N. Weiss,et al.  All-Uses versus Mutation Testing : An ExperimentalComparison of E ectiveness , 1996 .

[64]  Taghi M. Khoshgoftaar,et al.  How Many Software Metrics Should be Selected for Defect Prediction? , 2011, FLAIRS.

[65]  Boris Beizer,et al.  Black Box Testing: Techniques for Functional Testing of Software and Systems , 1996, IEEE Software.

[66]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[67]  Inderpal S. Bhandari,et al.  In-Process Improvement through Defect Data Interpretation , 1994, IBM Syst. J..

[68]  R. Natella,et al.  An Approach for Assessing Logs by Software Fault Injection , 2009 .

[69]  Rainer Unland,et al.  Objects, Components, Architectures, Services, and Applications for a Networked World , 2003, Lecture Notes in Computer Science.

[70]  Dewayne E. Perry,et al.  Software Faults in Evolving a Large, Real-Time System: a Case Study , 1993, ESEC.

[71]  Claes Wohlin,et al.  Determining the Improvement Potential of a Software Development Organization Through Fault Analysis: A Method and a Case Study , 2004, EuroSPI.

[72]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[73]  Rivalino Matias,et al.  An Experimental Study on Software Aging and Rejuvenation in Web Servers , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[74]  Paul A. Strooper,et al.  An Iterative Empirical Strategy for the Systematic Selection of a Combination of Verification and Validation Technologies , 2007, Fifth International Workshop on Software Quality (WoSQ'07: ICSE Workshops 2007).

[75]  Natalia Juristo Juzgado,et al.  Basics of Software Engineering Experimentation , 2010, Springer US.

[76]  Ram Chillarege,et al.  Orthogonal defect classification , 1996 .

[77]  Philip Koopman,et al.  The Exception Handling Effectiveness of POSIX Operating Systems , 2000, IEEE Trans. Software Eng..

[78]  Lars Lundberg,et al.  Identification of test process improvements by combining fault trigger classification and faults-slip-through measurement , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[79]  Elaine J. Weyuker,et al.  Metrics to Assess the Likelihood of Project Success Based on Architecture Reviews , 2004, Empirical Software Engineering.