Machine Learning Applied to Software Testing: A Systematic Mapping Study

Software testing involves probing into the behavior of software systems to uncover faults. Most testing activities are complex and costly, so a practical strategy that has been adopted to circumvent these issues is to automate software testing. There has been a growing interest in applying machine learning (ML) to automate various software engineering activities, including testing-related ones. In this paper, we set out to review the state-of-the art of how ML has been explored to automate and streamline software testing and provide an overview of the research at the intersection of these two fields by conducting a systematic mapping study. We selected 48 primary studies. These selected studies were then categorized according to study type, testing activity, and ML algorithm employed to automate the testing activity. The results highlight the most widely used ML algorithms and identify several avenues for future research. We found that ML algorithms have been used mainly for test-case generation, refinement, and evaluation. Also, ML has been used to evaluate test oracle construction and to predict the cost of testing-related activities. The results of this paper outline the ML algorithms that are most commonly used to automate software-testing activities, helping researchers to understand the current state of research concerning ML applied to software testing. We also found that there is a need for better empirical studies examining how ML algorithms have been used to automate software-testing activities.

[1]  Tao Xie,et al.  The synergy of human and artificial intelligence in software engineering , 2013, 2013 2nd International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE).

[2]  James M. Bieman,et al.  Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels , 2016, Softw. Test. Verification Reliab..

[3]  Aurora Trinidad Ramirez Pozo,et al.  Linking software testing results with a machine learning approach , 2013, Eng. Appl. Artif. Intell..

[4]  Upulee Kanewala,et al.  Using Semi-Supervised Learning for Predicting Metamorphic Relations , 2018, 2018 IEEE/ACM 3rd International Workshop on Metamorphic Testing (MET).

[5]  Roland Groz,et al.  Revisiting AI and Testing Methods to Infer FSM Models of Black-Box Systems , 2018, 2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST).

[6]  Mario Jino,et al.  Machine Learning Methods and Asymmetric Cost Function to Estimate Execution Effort of Software Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[7]  Farn Wang,et al.  Intelligent Test Oracle Construction for Reactive Systems without Explicit Specifications , 2011, 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing.

[8]  Jeffrey J. P. Tsai,et al.  Machine Learning and Software Engineering , 2002, 14th IEEE International Conference on Tools with Artificial Intelligence, 2002. (ICTAI 2002). Proceedings..

[9]  Tao Xie,et al.  Learning for test prioritization: an industrial case study , 2016, SIGSOFT FSE.

[10]  Joanna Strug,et al.  Machine Learning Approach in Mutation Testing , 2012, ICTSS.

[11]  Trevor Hastie,et al.  An Introduction to Statistical Learning , 2013, Springer Texts in Statistics.

[12]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[13]  Lionel C. Briand,et al.  Using machine learning to refine Category-Partition test specifications and test suites , 2009, Inf. Softw. Technol..

[14]  Domenico Cotroneo,et al.  A learning-based method for combining testing techniques , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[15]  Baowen Xu,et al.  Using semi-supervised clustering to improve regression test selection techniques , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[16]  Peter A. Flach,et al.  Machine Learning - The Art and Science of Algorithms that Make Sense of Data , 2012 .

[17]  Pearl Brereton,et al.  Evidence-Based Software Engineering and Systematic Reviews , 2015 .

[18]  Morten Mossige,et al.  Reinforcement learning for automatic test case prioritization and selection in continuous integration , 2017, ISSTA.

[19]  Emily Hill,et al.  Learning Effective Oracle Comparator Combinations for Web Applications , 2007, Seventh International Conference on Quality Software (QSIC 2007).

[20]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[21]  Pearl Brereton,et al.  Using mapping studies as the basis for further research - A participant-observer case study , 2011, Inf. Softw. Technol..

[22]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[23]  James V. Stone.  Bayes' Rule: A Tutorial Introduction to Bayesian Analysis , 2013 .

[24]  Francesco Bergadano,et al.  Testing by means of inductive program learning , 1996, TSEM.

[25]  Mourad Badri,et al.  Investigating the Accuracy of Test Code Size Prediction using Use Case Metrics and Machine Learning Algorithms: An Empirical Study , 2017, ICMLSC.

[26]  Abraham Kandel,et al.  Using a neural network in the software testing process , 2002, Int. J. Intell. Syst..

[27]  Jason Bell,et al.  Machine Learning: Hands-On for Developers and Technical Professionals , 2014 .

[28]  H. D. Rombach,et al.  The Goal Question Metric Approach , 1994 .

[29]  Abhishek Singhal,et al.  Generation of test oracles using neural network and decision tree model , 2014, 2014 5th International Conference - Confluence The Next Generation Information Technology Summit (Confluence).

[30]  Jungsoon P. Yoo,et al.  Software testing: a machine learning experiment , 1995, CSC '95.

[31]  Akbar Siami Namin,et al.  Bayesian reasoning for software testing , 2010, FoSER '10.

[32]  Mario Jino,et al.  Infeasible paths in the context of data flow based testing criteria: Identification, classification and prediction , 2010, Journal of the Brazilian Computer Society.

[33]  Gordon Fraser,et al.  Assessing and generating test sets in terms of behavioural adequacy , 2015, Softw. Test. Verification Reliab..

[34]  Du Zhang,et al.  Machine Learning in Value-Based Software Test Data Generation , 2006, 2006 18th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'06).

[35]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[36]  Gordon Fraser,et al.  Uncertainty-Driven Black-Box Test Data Generation , 2016, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[37]  Gregg Rothermel,et al.  Software testing: a research travelogue (2000–2014) , 2014, FOSE.

[38]  Robert Gove,et al.  Identifying Infeasible GUI Test Cases Using Support Vector Machines and Induced Grammars , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[39]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[40]  Anneliese Amschler Andrews,et al.  On the use of neural networks to guide software testing activities , 1995, Proceedings of 1995 IEEE International Test Conference (ITC).

[41]  James M. Rehg,et al.  Active learning for automatic classification of software behavior , 2004, ISSTA '04.

[42]  Li Hou,et al.  An Experience-Based Approach for Test Execution Effort Estimation , 2008, 2008 The 9th International Conference for Young Computer Scientists.

[43]  Gang Xiao,et al.  Software Testing by Active Learning for Commercial Games , 2005, AAAI.

[44]  Christopher A. Welty,et al.  Artificial Intelligence and Software Engineering: Breaking the Toy Mold , 2004, Automated Software Engineering.

[45]  Marlon Dumas,et al.  Browserbite: Accurate Cross-Browser Testing via Machine Learning over Image Features , 2013, 2013 IEEE International Conference on Software Maintenance.

[46]  F. Wotawa,et al.  Empirical Study of Correlation between Mutation Score and Model Inference Based Test Suite Adequacy Assessment , 2016, 2016 IEEE/ACM 11th International Workshop in Automation of Software Test (AST).

[47]  Jeremy S. Bradbury,et al.  Predicting mutation score using source code and test suite metrics , 2012, 2012 First International Workshop on Realizing AI Synergies in Software Engineering (RAISE).

[48]  Abraham Kandel,et al.  A Comparative Study of Artificial Neural Networks and Info-Fuzzy Networks as Automated Oracles in Software Testing , 2012, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[49]  Ebrahim Bagheri,et al.  Machine Learning-based Software Testing: Towards a Classification Framework , 2011, SEKE.

[50]  T. H. Tse,et al.  Finding failures from passed test cases: improving the pattern classification approach to the testing of mesh simplification programs , 2010 .

[51]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[52]  Panagiotis Louridas,et al.  Machine Learning , 2016, IEEE Software.

[53]  Mark Harman,et al.  The role of Artificial Intelligence in Software Engineering , 2012, 2012 First International Workshop on Realizing AI Synergies in Software Engineering (RAISE).

[54]  Lionel C. Briand.  Novel Applications of Machine Learning in Software Testing , 2008, 2008 The Eighth International Conference on Quality Software.

[55]  Paolo Tonella,et al.  Using the Case-Based Ranking Methodology for Test Case Prioritization , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[56]  Mary Jean Harrold,et al.  Testing: a roadmap , 2000, ICSE '00.

[57]  Paul E. Ceruzzi,et al.  Computing: A Concise History , 2012 .

[58]  Paulo Tabuada,et al.  Underminer , 2017, ACM Trans. Embed. Comput. Syst..

[59]  Frits W. Vaandrager,et al.  Improving active Mealy machine learning for protocol conformance testing , 2014, Machine Learning.

[60]  Ariel Rosenfeld,et al.  Automation of Android Applications Functional Testing Using Machine Learning Activities Classification , 2018, 2018 IEEE/ACM 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft).

[61]  Alper Sen,et al.  Testing Service Oriented Architectures Using Stateful Service Virtualization via Machine Learning , 2018, 2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST).

[62]  Muhammad Ali Babar,et al.  Identifying relevant studies in software engineering , 2011, Inf. Softw. Technol..

[63]  Leonardo Mariani,et al.  Automatic testing of GUI‐based applications , 2014, Softw. Test. Verification Reliab..

[64]  Mirko Conrad,et al.  Test case design based on Z and the classification-tree method , 1997, First IEEE International Conference on Formal Engineering Methods.

[65]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[66]  Brett Lantz,et al.  Machine learning with R : learn how to use R to apply powerful machine learning methods and gain an insight into real-world applications , 2013 .

[67]  Roel Wieringa,et al.  Requirements engineering paper classification and evaluation criteria: a proposal and a discussion , 2005, Requirements Engineering.

[68]  Elaine J. Weyuker,et al.  Data flow analysis techniques for test data selection , 2015, ICSE '82.

[69]  Kai Petersen,et al.  Guidelines for conducting systematic mapping studies in software engineering: An update , 2015, Inf. Softw. Technol..

[70]  Hardi Hungar,et al.  Domain-Specific Optimization in Automata Learning , 2003, CAV.

[71]  Lloyd G. Greenwald,et al.  An exploration of statistical models for automated test case generation , 2005, ACM SIGSOFT Softw. Eng. Notes.

[72]  Lu Zhang,et al.  Predictive Mutation Testing , 2016, IEEE Transactions on Software Engineering.

[73]  Yi Wang,et al.  Artificial Neural Network for Automatic Test Oracles Generation , 2008, 2008 International Conference on Computer Science and Software Engineering.

[74]  J. Christopher Westland,et al.  The cost of errors in software development: evidence from industry , 2002, J. Syst. Softw..

[75]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[76]  George C. Necula,et al.  Guided GUI testing of android apps with minimal restart and approximate learning , 2013, OOPSLA.

[77]  Michael Bowles.  Machine Learning in Python: Essential Techniques for Predictive Analysis , 2015 .

[78]  Petra Kaufmann,et al.  Experimental And Quasi Experimental Designs For Research , 2016 .

[79]  Harald C. Gall,et al.  How high will it be? Using machine learning models to predict branch coverage in automated testing , 2018, 2018 IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE).