A machine learning approach to generate test oracles

One of the essential activities for quality assurance in software development is the software testing. Studies report that Software Testing is one of the most costly activities in the development process, can reach up to 50 percent of its total cost. One of the great challenges of conducting software testing is related to the automation of a mechanism known as "test oracle". This work presents an approach based on machine learning (ML) for automation of the test oracle mechanism in software. The approach uses historical usage data from an application captured by inserting a capture component into the application under test. These data go through a Knowledge Discovery in Database step and are then used for training to generate an oracle suitable for the application under test. Four experiments were executed with web applications to evaluate the proposed approach. The first and second experiments were performed with a fictitious application, with faults inserted randomly in the first experiment, inserted by a developer in the second one and inserted by mutation tests in third one. The fourth experiment was carried out with a large real application in order to assure the results of the preliminary experiments. The experiments presented indications of the suitability of the approach to the solution of the problem.

[1]  Ali Mesbah,et al.  PYTHIA: Generating test cases with oracles for JavaScript applications , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[2]  Browne,et al.  Cross-Validation Methods. , 2000, Journal of mathematical psychology.

[3]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[4]  Heikki Mannila,et al.  A database perspective on knowledge discovery , 1996, CACM.

[5]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[6]  Michael Stonebraker,et al.  Database systems: achievements and opportunities , 1990, SGMD.

[7]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[8]  Soila Pertet,et al.  Causes of Failure in Web Applications (CMU-PDL-05-109) , 2005 .

[9]  T. H. Tse,et al.  Oracles Are Hardly Attain'd, and Hardly Understood: Confessions of Software Testing Researchers , 2013, 2013 13th International Conference on Quality Software.

[10]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[11]  J. Kent.  Information gain and a general measure of correlation , 1983 .

[12]  Gregory Gay,et al.  Automated Oracle Data Selection Support , 2015, IEEE Transactions on Software Engineering.

[13]  Connolly,et al.  Database Systems , 2004 .

[14]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[15]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[16]  Xin Zhang,et al.  Monic Testing of Web Services Based on Algebraic Specifications , 2016, 2016 IEEE Symposium on Service-Oriented System Engineering (SOSE).

[17]  Patrícia Duarte de Lima Machado.  Testing from structured algebraic specifications : the oracle problem , 2000 .

[18]  Usama M. Fayyad,et al.  Knowledge Discovery in Databases: An Overview , 1997, ILP.

[19]  Dongmei Zhang,et al.  A Machine Learning Approach for Developing Test Oracles for Testing Scientific Software , 2016, SEKE.

[20]  Shing-Chi Cheung,et al.  Towards a metamorphic testing methodology for service-oriented software applications , 2005, Fifth International Conference on Quality Software (QSIC'05).

[21]  Johannes Fürnkranz,et al.  Incremental Reduced Error Pruning , 1994, ICML.

[22]  Chris Thornton,et al.  Auto-WEKA : combined selection and hyperparameter optimization of supervised machine learning algorithms , 2014 .

[23]  Muhammad Zohaib Z. Iqbal,et al.  An automated model based testing approach for platform games , 2015, 2015 ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS).

[24]  Ingolf Geist,et al.  A framework for data mining and KDD , 2002, SAC '02.

[25]  Siti Zaiton Mohd Hashim,et al.  Artificial neural networks as multi-networks automated test oracle , 2011, Automated Software Engineering.

[26]  Upulee Kanewala,et al.  Automated Test Oracles: State of the Art, Taxonomies, and Trends , 2015, Adv. Comput..

[27]  Farn Wang,et al.  Evolving a Test Oracle in Black-Box Testing , 2011, FASE.

[28]  Suhaimi Ibrahim,et al.  A Single-Network ANN-based Oracle to verify logical software modules , 2010, 2010 2nd International Conference on Software Technology and Engineering.

[29]  G. Myers,et al.  The Art of Software Testing: Myers/Art , 2012 .

[30]  C. I. Mosier.  I. Problems and Designs of Cross-Validation 1 , 1951 .

[31]  Luiz Eduardo Soares de Oliveira,et al.  Combining overall and local class accuracies in an oracle-based method for dynamic ensemble selection , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[32]  Márcio Eduardo Delamaro,et al.  Using concepts of content‐based image retrieval to implement graphical testing oracles , 2013, Softw. Test. Verification Reliab..

[33]  Gordon Fraser,et al.  1600 faults in 100 projects: automatically finding faults while achieving high coverage with EvoSuite , 2015, Empirical Software Engineering.

[34]  Jiawei Han,et al.  Knowledge Discovery in Databases: An Attribute-Oriented Approach , 1992, VLDB.

[35]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[36]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[37]  Harvey P. Siy,et al.  Semantics-based Automated Web Testing , 2015, WWV.

[38]  Richard G. Hamlet,et al.  Software Quality, Software Process, and Software Testing , 1995, Adv. Comput..

[39]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[40]  Siti Zaiton Mohd Hashim,et al.  A Comparative Study on Automated Software Test Oracle Methods , 2009, 2009 Fourth International Conference on Software Engineering Advances.