A Comprehensive Survey of Trends in Oracles for Software Testing

Testing involves examining the behaviour of a system in order to discover potential faults. Determining the desired correct behaviour for a given input is called the “oracle problem”. Oracle automation is important to remove a current bottleneck which inhibits greater overall test automation; without oracle automation, the human has to determine whether observed behaviour is correct. The literature on oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of oracle, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the oracle problem and an analysis of trends in this important area of software testing research and practice.

[1]  Frederick C. Hennie,et al.  Finite-state Models for Logical Machines , 1968 .

[2]  Andrzej Blikle.  Proving Programs by Sets of Computations , 1974, MFCS.

[3]  Robert M. Keller,et al.  Formal verification of parallel programs , 1976, CACM.

[4]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[5]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[6]  Gregor von Bochmann,et al.  Finite State Description of Communication Protocols , 1978, Comput. Networks.

[7]  William E. Howden,et al.  Theoretical and Empirical Studies of Program Testing , 1978, IEEE Transactions on Software Engineering.

[8]  Alan F. Babich.  Proving Total Correctness of Parallel Programs , 1979, IEEE Transactions on Software Engineering.

[9]  Elaine J. Weyuker.  Translatability and Decidability Questions for Restricted Classes of Program Schemas , 1979, SIAM J. Comput..

[10]  Joseph Sifakis,et al.  Deadlocks and Livelocks in Transition Systems , 1980, MFCS.

[11]  Richard G. Hamlet,et al.  Data Abstraction, Implementation, Specification, and Testing , 1981, TOPL.

[12]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[13]  Elaine J. Weyuker,et al.  Assessing Test Data Adequacy through Program Inference , 2019, TOPL.

[14]  An approach to testing specifications , 1983, SIGSOFT '83.

[15]  David C. Luckham,et al.  An Overview of Anna, a Specification Language for Ada , 1985, IEEE Software.

[16]  Algirdas Avizienis,et al.  The N-Version Approach to Fault-Tolerant Software , 1985, IEEE Transactions on Software Engineering.

[17]  William E. Howden,et al.  A functional approach to program testing and analysis , 1986, IEEE Transactions on Software Engineering.

[18]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[19]  Dana Angluin,et al.  Learning Regular Sets from Queries and Counterexamples , 1987, Inf. Comput..

[20]  Mike Holcombe.  X-machines as a basis for dynamic system specification , 1988, Softw. Eng. J..

[21]  Ben Zorn,et al.  A memory allocation profiler for c and lisp , 1988 .

[22]  Bertrand Meyer,et al.  Eiffel: A language and environment for software engineering , 1988, J. Syst. Softw..

[23]  Nancy A. Lynch,et al.  An introduction to input/output automata , 1989 .

[24]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[25]  A. M. Turing,et al.  Checking a large routine , 1989 .

[26]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[27]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[28]  Jeannette M. Wing.  A specifier's introduction to formal methods , 1990, Computer.

[29]  Jingde Cheng.  A survey of tasking deadlock detection methods , 1991, ALET.

[30]  A.P. Mathur.  Performance, effectiveness, and reliability issues in software testing , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[31]  Gilles Bernot,et al.  Testing Against Formal Specifications: A Theoretical View , 1991, TAPSOFT, Vol.2.

[32]  Marie-Claude Gaudel,et al.  Software testing based on formal specifications: a theory and a tool , 1991, Softw. Eng. J..

[33]  Jeffrey M. Voas,et al.  PIE: A Dynamic Failure-Based Technique , 1992, IEEE Trans. Software Eng..

[34]  David Lorge Parnas,et al.  Trace Rewriting Systems , 1992, CTRS.

[35]  Stephen J. Garland,et al.  An Overview of Larch , 1993, Functional Programming, Concurrency, Simulation and Automated Reasoning.

[36]  Richard A. Eyre-Todd.  The detection of dangling references in C++ programs , 1993, LOPL.

[37]  Robert W. Floyd,et al.  Assigning Meanings to Programs , 1993 .

[38]  David Lorge Parnas,et al.  Simulating the behaviour of software modules by trace rewriting , 1993, ICSE '93.

[39]  Todd M. Austin,et al.  Efficient detection of all pointer and array access errors , 1994, PLDI '94.

[40]  Alexandre Petrenko,et al.  Protocol testing: review of methods and relevance for software testing , 1994, ISSTA '94.

[41]  Umesh V. Vazirani,et al.  An Introduction to Computational Learning Theory , 1994 .

[42]  David Lorge Parnas,et al.  Precise Documentation of Well-Structured Programs , 1994, IEEE Trans. Software Eng..

[43]  Phyllis G. Frankl,et al.  The ASTOOT approach to testing object-oriented programs , 1994, TSEM.

[44]  David Lorge Parnas,et al.  Functional Documents for Computer Systems , 1995, Sci. Comput. Program..

[45]  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[46]  Bruno Marre,et al.  LOFT: A Tool for Assisting Selection of Test Data Sets from Algebraic Specifications , 1995, TAPSOFT.

[47]  David S. Rosenblum.  A Practical Approach to Programming With Assertions , 1995, IEEE Trans. Software Eng..

[48]  Kevin Lano,et al.  Specification in B: An Introduction Using the B Toolkit , 1996 .

[49]  P. David Stotts,et al.  Daistish: systematic algebraic testing for OO programs in the presence of side-effects , 1996, ISSTA '96.

[50]  Jan Tretmans,et al.  Test Generation with Inputs, Outputs and Repetitive Quiescence , 1996, Softw. Concepts Tools.

[51]  John R. Callahan,et al.  Automated Software Testing Using Model-Checking , 1996 .

[52]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[53]  Bogdan Korel,et al.  The chaining approach for software test data generation , 1996, TSEM.

[54]  Dieter Hogrefe,et al.  SDL : formal object-oriented language for communicating systems , 1997 .

[55]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[56]  Felix T.S. Chan,et al.  Application of metamorphic testing in numerical analysis , 1998, ICSE 1998.

[57]  Florentin Ipate,et al.  Correct systems - building a business process solution , 1998, Applied computing.

[58]  Marie-Claude Gaudel,et al.  Testing Algebraic Data Types and Processes: A Unifying Theory , 1998, Formal Aspects of Computing.

[59]  Daniel P. Siewiorek,et al.  Automated robustness testing of off-the-shelf software components , 1998, Digest of Papers. Twenty-Eighth Annual International Symposium on Fault-Tolerant Computing (Cat. No.98CB36224).

[60]  Egon Börger,et al.  High Level System Design and Analysis Using Abstract State Machines , 1998, FM-Trends.

[61]  Peter Gorm Larsen,et al.  Modelling Systems - Practical Tools and Techniques in Software Development (2. ed.) , 1998 .

[62]  Tsong Yueh Chen,et al.  In black and white: an integrated approach to class-level testing of object-oriented programs , 1998, TSEM.

[63]  허윤정,et al.  Holzmann의 ˝The Model Checker SPIN˝에 대하여 , 1998 .

[64]  David Lorge Parnas,et al.  Using Test Oracles Generated from Program Documentation , 1998, IEEE Trans. Software Eng..

[65]  Robert Feldt,et al.  Generating diverse software versions with genetic programming: and experimental study , 1998, IEE Proc. Softw..

[66]  Patrícia Duarte de Lima Machado.  On Oracles for Interpreting Test Results against Algebraic Specifications , 1998, AMAST.

[67]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[68]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[69]  Bernhard K. Aichernig,et al.  Automated Black-Box Testing with Abstract VDM Oracles , 1999, SAFECOMP.

[70]  Mihalis Yannakakis,et al.  Black Box Checking , 1999, FORTE.

[71]  Mary Lou Soffa,et al.  Automated test oracles for GUIs , 2000, SIGSOFT '00/FSE-8.

[72]  Elvinia Riccobene,et al.  Modeling the Dynamics of UML State Machines , 2000, Abstract State Machines.

[73]  Gregg Rothermel,et al.  An empirical investigation of the relationship between spectra differences and regression faults , 2000, Softw. Test. Verification Reliab..

[74]  C. J. Harwood.  Correct Systems: Building a Business Process Solution , 2000 .

[75]  David Lorge Parnas,et al.  Requirements-based monitors for real-time systems , 2000, ISSTA '00.

[76]  Elaine J. Weyuker,et al.  Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study , 2000, IEEE Trans. Software Eng..

[77]  Axel van Lamsweerde,et al.  Formal specification: a roadmap , 2000, ICSE '00.

[78]  Laura K. Dillon,et al.  Automated support for testing and debugging of real-time programs using oracles , 2000, SOEN.

[79]  K. Rustan M. Leino,et al.  Houdini, an Annotation Assistant for ESC/Java , 2001, FME.

[80]  Debbie Richards,et al.  Towards Reconciling Use Cases via Controlled Language and Graphical Models , 2001, INAP.

[81]  Tsong Yueh Chen,et al.  TACCLE: a methodology for object-oriented software testing at the class and cluster levels , 2001, TSEM.

[82]  Marie-Claude Gaudel,et al.  Testing from Formal Specifications, a Generic Approach , 2001, Ada-Europe.

[83]  M. Young.  Test Oracles , 2001 .

[84]  A. Tiwari.  Formal Semantics and Analysis Methods for Simulink Stateflow Models , 2001 .

[85]  Angelo Gargantini,et al.  ASM-Based Testing: Coverage Criteria and Automatic Test Sequence , 2001, J. Univers. Comput. Sci..

[86]  Rolf Schwitter,et al.  English as a formal specification language , 2002, Proceedings. 13th International Workshop on Database and Expert Systems Applications.

[87]  Gary T. Leavens,et al.  A Simple and Practical Approach to Unit Testing: The JML and JUnit Way , 2002, ECOOP.

[88]  Gregg Rothermel,et al.  Empirical studies of test‐suite reduction , 2002, Softw. Test. Verification Reliab..

[89]  Peter D. Mosses,et al.  CASL: the Common Algebraic Specification Language , 2002, Theor. Comput. Sci..

[90]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[91]  Tsong Yueh Chen,et al.  Metamorphic testing of programs on partial differential equations: a case study , 2002, Proceedings 26th Annual International Computer Software and Applications.

[92]  Alex Groce,et al.  AMC: An Adaptive Model Checker , 2002, CAV.

[93]  Monica S. Lam,et al.  A practical flow-sensitive and context-sensitive C and C++ memory leak detector , 2003, PLDI '03.

[94]  David Notkin,et al.  Mutually Enhancing Test Generation and Specification Inference , 2003, FATES.

[95]  Hong Zhu.  A note on test oracles and semantics of algebraic specifications , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[96]  Tsong Yueh Chen,et al.  Metamorphic testing and beyond , 2003, Eleventh Annual International Workshop on Software Technology and Engineering Practice.

[97]  Stacy J. Prowell,et al.  Foundations of Sequence-Based Software Specification , 2003, IEEE Trans. Software Eng..

[98]  Amer Diwan,et al.  Discovering Algebraic Specifications from Java Classes , 2003, ECOOP.

[99]  Hong Sun,et al.  Investigating the use of analysis contracts to improve the testability of object‐oriented code , 2003, Softw. Pract. Exp..

[100]  Michel Sintzoff,et al.  Formal derivation of strongly correct concurrent programs , 2004, Acta Informatica.

[101]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[102]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[103]  Using transient/persistent errors to develop automated test oracles for event-driven software , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[104]  David Notkin,et al.  Checking inside the black box: regression testing by comparing value spectra , 2005, IEEE Transactions on Software Engineering.

[105]  David Lorge Parnas,et al.  Using information about functions in selecting test cases , 2005, A-MOST '05.

[106]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[107]  Shing-Chi Cheung,et al.  Towards a metamorphic testing methodology for service-oriented software applications , 2005, Fifth International Conference on Quality Software (QSIC'05).

[108]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[109]  Tsong Yueh Chen,et al.  A metamorphic approach to integration testing of context-sensitive middleware-based applications , 2005, Fifth International Conference on Quality Software (QSIC'05).

[110]  Thomas A. Henzinger,et al.  Checking Memory Safety with Blast , 2005, FASE.

[111]  Michael D. Ernst,et al.  Eclat: Automatic Generation and Classification of Test Inputs , 2005, ECOOP.

[112]  David Coppit,et al.  On the Use of Specification-Based Assertions as Test Oracles , 2005, 29th Annual IEEE/NASA Software Engineering Workshop.

[113]  Daniel Jackson,et al.  Software Abstractions - Logic, Language, and Analysis , 2006 .

[114]  Paolo Tonella,et al.  Detecting anomaly and failure in Web applications , 2006, IEEE Multimedia.

[115]  Stephen S. Yau,et al.  Integration Testing of Context-sensitive Middleware-based Applications: a Metamorphic Approach , 2006, Int. J. Softw. Eng. Knowl. Eng..

[116]  Ying Li,et al.  A survey of systems for detecting serial run‐time errors , 2006, Concurr. Comput. Pract. Exp..

[117]  Neelam Gupta,et al.  Test Case Prioritization Using Relevant Slices , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[118]  Mark Harman,et al.  Automated Unique Input Output Sequence Generation for Conformance Testing of FSMs , 2006, Comput. J..

[119]  Edith Elkind,et al.  Grey-Box Checking , 2006, FORTE.

[120]  Tao Xie,et al.  Augmenting Automatically Generated Unit-Test Suites with Regression Oracle Checking , 2006, ECOOP.

[121]  Michael D. Ernst,et al.  Finding the needles in the haystack: Generating legal test inputs for object-oriented programs , 2006 .

[122]  Keqin Li,et al.  Integration Testing of Components Guided by Incremental State Machine Learning , 2006, Testing: Academic & Industrial Conference - Practice And Research Techniques (TAIC PART'06).

[123]  Paul C. Clements.  Managing Variability for Software Product Lines: Working with Variability Mechanisms , 2006, 10th International Software Product Line Conference (SPLC'06).

[124]  M. Ben-Ari,et al.  Principles of Concurrent and Distributed Programming (2nd Edition) (Prentice-Hall International Series in Computer Science) , 2006 .

[125]  Atif M. Memon,et al.  Designing and comparing automated test oracles for GUI-based software applications , 2007, TSEM.

[126]  W. Chan,et al.  A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications , 2007, Int. J. Web Serv. Res..

[127]  Xin Feng,et al.  Tabular Expression-Based Testing Strategies: A Comparison , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[128]  Fabrice Bouquet,et al.  A subset of precise UML for model-based testing , 2007, A-MOST '07.

[129]  Johannes Mayer,et al.  Statistical Metamorphic Testing Testing Programs with Random Output by Means of Statistical Hypothesis Tests and Metamorphic Testing , 2007, Seventh International Conference on Quality Software (QSIC 2007).

[130]  K. Bogdanov,et al.  A comparative study of methods for dynamic reverse-engineering of state models , 2007 .

[131]  Andreas Zeller,et al.  Efficient unit test case minimization , 2007, ASE '07.

[132]  Stephen McCamant,et al.  The Daikon system for dynamic detection of likely invariants , 2007, Sci. Comput. Program..

[133]  Anthony J. H. Simons,et al.  JWalk: a tool for lazy, systematic testing of java classes by design introspection and user interaction , 2007, Automated Software Engineering.

[134]  Abstraction in Assertion-Based Test Oracles , 2007, Seventh International Conference on Quality Software (QSIC 2007).

[135]  Jared D. DeMott,et al.  Fuzzing for Software Security Testing and Quality Assurance , 2008 .

[136]  Dawson R. Engler,et al.  EXE: Automatically Generating Inputs of Death , 2008, TSEC.

[137]  Neil Walkinshaw,et al.  Inferring Finite-State Models with Temporal Constraints , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[138]  David Lee,et al.  Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning , 2008, FORTE.

[139]  Leonardo Mariani,et al.  Automatic generation of software behavioral models , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[140]  Keqin Li,et al.  Modular System Verification by Inference, Testing and Reachability Analysis , 2008, TestCom/FATES.

[141]  Atif M. Memon,et al.  Automatically repairing event sequence-based GUI test suites for regression testing , 2008, TSEM.

[142]  Bo Yu,et al.  Testing Java Components based on Algebraic Specifications , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[143]  Xin Yao,et al.  Search based software testing of object-oriented containers , 2008, Inf. Sci..

[144]  Mark Harman,et al.  Automated Session Data Repair for Web Application Regression Testing , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[145]  John Derrick,et al.  Iterative Refinement of Reverse-Engineered Models by Model-Based Testing , 2009, FM.

[146]  Robert M. Hierons.  Verdict functions in testing with a fault domain or test hypotheses , 2009, TSEM.

[147]  David Lorge Parnas,et al.  Document based rational software development , 2009, Knowl. Based Syst..

[148]  Mark Harman,et al.  Regression Testing Minimisation, Selection and Prioritisation - A Survey , 2009 .

[149]  Phil McMinn,et al.  Search-based failure discovery using testability transformations to generate pseudo-oracles , 2009, GECCO.

[150]  Paola Inverardi,et al.  Automatic synthesis of behavior protocols for composable web-services , 2009, ESEC/FSE '09.

[151]  Gail E. Kaiser,et al.  Automatic system testing of programs without test oracles , 2009, ISSTA.

[152]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[153]  Yuanyuan Zhang,et al.  Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications , 2009 .

[154]  Gail E. Kaiser,et al.  Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles , 2009, 2009 International Conference on Software Testing Verification and Validation.

[155]  Rance Cleaveland,et al.  Using formal specifications to support testing , 2009, CSUR.

[156]  Bertrand Meyer,et al.  A comparative study of programmer-written and automatically inferred contracts , 2009, ISSTA.

[157]  Siti Zaiton Mohd Hashim,et al.  A Comparative Study on Automated Software Test Oracle Methods , 2009, 2009 Fourth International Conference on Software Engineering Advances.

[158]  Rishabh Singh,et al.  Learning Component Interfaces with May and Must Abstractions , 2010, CAV.

[159]  Mark Harman,et al.  Reducing qualitative human oracle costs associated with automatically generated test data , 2010, STOV '10.

[160]  Myra B. Cohen,et al.  Directed test suite augmentation: techniques and tradeoffs , 2010, FSE '10.

[161]  Mark Harman,et al.  n empirical investigation into branch coverage for C programs using CUTE and USTIN , 2010 .

[162]  David Lorge Parnas,et al.  Precise Documentation: The Key to Better Software , 2010, The Future of Software Engineering.

[163]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[164]  Mark Harman,et al.  Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[165]  Shin Yoo.  Metamorphic Testing of Stochastic Optimisation , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[166]  David Lorge Parnas,et al.  Defining the meaning of tabular mathematical expressions , 2010, Sci. Comput. Program..

[167]  John Derrick,et al.  Increasing Functional Coverage by Inductive Testing: A Case Study , 2010, ICTSS.

[168]  Lionel C. Briand,et al.  On the Effectiveness of Contracts as Test Oracles in the Detection and Diagnosis of Race Conditions and Deadlocks in Concurrent Object-Oriented Software , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[169]  Michael D. Ernst,et al.  Combined static and dynamic automated test generation , 2011, ISSTA '11.

[170]  Lionel C. Briand,et al.  Enabling the runtime assertion checking of concurrent contracts for the Java modeling language , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[171]  John A. Clark,et al.  Searching for invariants using genetic programming and mutation testing , 2011, GECCO '11.

[172]  Phil McMinn,et al.  Search-Based Software Testing: Past, Present and Future , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[173]  Mark Harman,et al.  Automated web application testing using search based software engineering , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[174]  Sergio Segura,et al.  Automated metamorphic testing on the analyses of feature models , 2011, Inf. Softw. Technol..

[175]  Mats Per Erik Heimdahl,et al.  Programs, tests, and oracles: the foundations of testing revisited , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[176]  Bertrand Meyer,et al.  Stateful testing: Finding more errors in code and contracts , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[177]  Paolo Tonella,et al.  Symbolic search-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[178]  Leonardo Mariani,et al.  Dynamic Analysis for Diagnosing Integration Faults , 2011, IEEE Transactions on Software Engineering.

[179]  Siti Zaiton Mohd Hashim,et al.  An automated framework for software test oracle , 2011, Inf. Softw. Technol..

[180]  Bertrand Meyer,et al.  Inferring better contracts , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[181]  Mark Harman,et al.  Automatically generating realistic test input from web services , 2011, Proceedings of 2011 IEEE 6th International Symposium on Service Oriented System (SOSE).

[182]  David Lorge Parnas,et al.  A Comparison of Tabular Expression-Based Testing Strategies , 2011, IEEE Transactions on Software Engineering.

[183]  Siti Zaiton Mohd Hashim,et al.  Artificial neural networks as multi-networks automated test oracle , 2011, Automated Software Engineering.

[184]  Andreas Zeller,et al.  Exploiting Common Object Usage in Test Case Generation , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[185]  Massimiliano Di Penta,et al.  Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments , 2011, IEEE Transactions on Software Engineering.

[186]  Tao Xie,et al.  Mining test oracles of web search engines , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[187]  Matthew J. Rutherford,et al.  An Empirical Evaluation of Assertions as Oracles , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[188]  Andreas Zeller,et al.  Assessing Oracle Quality with Checked Coverage , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[189]  Leo Grady,et al.  Automating image segmentation verification and validation by learning test oracles , 2011, Inf. Softw. Technol..

[190]  Roland Groz,et al.  Finding Software Vulnerabilities by Smart Fuzzing , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[191]  Tiziana Margaria,et al.  Leveraging Applications of Formal Methods, Verification and Validation. Technologies for Mastering Change , 2012, Lecture Notes in Computer Science.

[192]  Bruno Legeard,et al.  A taxonomy of model‐based testing approaches , 2012, Softw. Test. Verification Reliab..

[193]  Robert M. Hierons.  Oracles for Distributed Testing , 2012, IEEE Transactions on Software Engineering.

[194]  Bernhard Steffen,et al.  Automated Inference of Models for Black Box Systems Based on Interface Descriptions , 2012, ISoLA.

[195]  Tsong Yueh Chen,et al.  Automated functional testing of online search services , 2012, Softw. Test. Verification Reliab..

[196]  Gregory Gay,et al.  Automated oracle creation support, or: How I learned to stop worrying about fault propagation and love mutation testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[197]  Marijn J. H. Heule,et al.  Software model synthesis using satisfiability solvers , 2012, Empirical Software Engineering.

[198]  Enrique Alba,et al.  Evolutionary algorithms for the multi‐objective test data generation problem , 2012, Softw. Pract. Exp..

[199]  Gregg Rothermel,et al.  Oracle-Centric Test Case Prioritization , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[200]  Mathew Hall,et al.  Using Behaviour Inference to Optimise Regression Test Sets , 2012, ICTSS.

[201]  Gregg Rothermel,et al.  Understanding user understanding: determining correctness of generated program invariants , 2012, ISSTA 2012.

[202]  Phil McMinn,et al.  Search-Based Test Input Generation for String Data Types Using the Results of Web Queries , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[203]  Christian Bird,et al.  Assessing the value of branches with what-if analysis , 2012, SIGSOFT FSE.

[204]  Gordon Fraser,et al.  CrowdOracles: Can the Crowd Solve the Oracle Problem? , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[205]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[206]  Sheeva Afshan,et al.  Evolving Readable String Test Inputs Using a Natural Language Model to Reduce Human Oracle Cost , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[207]  Lauretta O. Osho,et al.  Axiomatic Basis for Computer Programming , 2013 .

[208]  Cliff B. Jones,et al.  Turing's "Checking a Large Routine" , 2013 .

[209]  Ahmed E. Hassan,et al.  Automatic detection of performance deviations in the load testing of Large Scale Systems , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[210]  Peter Kulchyski.  and , 2015 .