A systematic literature review on semantic web enabled software testing

Abstract Software testing, as a major verification and validation activity which revolves around quality tests, is a knowledge-intensive activity. Hence, it is reasonable to expect that it can be improved by effective application of semantic web technologies, e.g., ontologies, which have been frequently used in knowledge engineering activities. The objective of this work is to investigate and provide a better understanding of how semantic web enabled techniques, i.e., the techniques that are based on the effective application of the semantic web technologies, have been used to support software testing activities. For this purpose, a Systematic Literature Review based on a predefined procedure is conducted. A total of 52 primary studies were identified as relevant, which have undergone a thorough meta-analysis with regards to our posed research questions. This study indicates the benefits of semantic web enabled software testing in both industry and academia. It also identifies main software testing activities that can benefit from the semantic web enabled techniques. Furthermore, contributions of such techniques to the testing process are thoroughly examined. Finally, potentials and difficulties of applying these techniques to software testing, along with the promising research directions are discussed.

[1]  Giancarlo Guizzardi,et al.  Organizing Ontology Design Patterns as Ontology Pattern Languages , 2013, ESWC.

[2]  Xuexiang Li,et al.  Ontology-Based Testing Platform for Reusing , 2012, 2012 Sixth International Conference on Internet Computing for Science and Engineering.

[3]  Yinong Chen,et al.  Automatic Mutation Testing and Simulation on OWL-S Specified Web Services , 2008, 41st Annual Simulation Symposium (anss-41 2008).

[4]  Deborah L. McGuinness,et al.  PROV-O: The PROV Ontology , 2013 .

[5]  Zhenyu Liu,et al.  Test Case Reuse Based on Ontology , 2009, 2009 15th IEEE Pacific Rim International Symposium on Dependable Computing.

[6]  Leonardo Mariani,et al.  Link: exploiting the web of data to generate test inputs , 2014, ISSTA 2014.

[7]  Rui Li,et al.  The implementation of user interface autogenerate for spacecraft automatic tests based on ontology , 2015, 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).

[8]  Thomas R. Gruber,et al.  Toward principles for the design of ontologies used for knowledge sharing? , 1995, Int. J. Hum. Comput. Stud..

[9]  Vladan Devedzic,et al.  Converting UML to OWL ontologies , 2004, WWW Alt. '04.

[10]  Ricardo de Almeida Falbo,et al.  Knowledge management initiatives in software testing: A mapping study , 2015, Inf. Softw. Technol..

[11]  Ahmet Coşkunçay,et al.  An Ontology to Support TMMi-Based Test Process Assessment , 2018 .

[12]  Mohsen Kahani,et al.  An automated model based approach to test web application using ontology , 2011, 2011 IEEE Conference on Open Systems.

[13]  Birgit Vogel-Heuser,et al.  Applications of Semantic Web Technologies for the Engineering of Automated Production Systems - Three Use Cases , 2016, Semantic Web Technologies for Intelligent Engineering Applications.

[14]  Ali Selamat,et al.  A systematic literature review of software requirements prioritization research , 2014, Inf. Softw. Technol..

[15]  Mohsen Kahani,et al.  Ontology-Based Web Application Testing , 2008, TeNe.

[16]  Hrushikesha Mohanty,et al.  An Ontology Based Approach for Test Scenario Management , 2011, ICISTM.

[17]  Hong Zhu,et al.  A Test Automation Framework for Collaborative Testing of Web Service Dynamic Compositions , 2014, Advanced Web Services.

[18]  Usman Qamar,et al.  Ontology Based Test Case Generation for Black Box Testing , 2019 .

[19]  Paris Avgeriou,et al.  Variability in quality attributes of service-based software systems: A systematic literature review , 2013, Inf. Softw. Technol..

[20]  Abdul Rauf,et al.  Ontology driven semantic annotation based GUI testing , 2010, 2010 6th International Conference on Emerging Technologies (ICET).

[21]  Chin-Feng Fan,et al.  Validation test case generation based on safety analysis ontology , 2012 .

[22]  Ricardo de Almeida Falbo,et al.  Semantic Documentation in Requirements Engineering , 2014, WER.

[23]  Elisa Yumi Nakagawa,et al.  Exploring ontologies to support the establishment of reference architectures: An example on software testing , 2009, 2009 Joint Working IEEE/IFIP Conference on Software Architecture & European Conference on Software Architecture.

[24]  Gleidson Bertollo,et al.  Establishing a Common Vocabulary for Software Organizations Understand Software Processes , 2005 .

[25]  Ajitha Rajan,et al.  CESAR - Cost-efficient Methods and Processes for Safety-relevant Embedded Systems , 2013 .

[26]  Mika Mäntylä,et al.  The Role of the Tester's Knowledge in Exploratory Software Testing , 2013, IEEE Transactions on Software Engineering.

[27]  Wei-Tek Tsai,et al.  Ontology-Based Test Modeling and Partition Testing of Web Services , 2008, 2008 IEEE International Conference on Web Services.

[28]  Benjamin Michotte,et al.  USIXML: A Language Supporting Multi-path Development of User Interfaces , 2004, EHCI/DS-VIS.

[29]  Pavel Vrba MAST: manufacturing agent simulation tool , 2003, EFTA 2003. 2003 IEEE Conference on Emerging Technologies and Factory Automation. Proceedings (Cat. No.03TH8696).

[30]  Jun Wang,et al.  An Ontology Based Improved Software Requirement Traceability Matrix , 2009, 2009 Second International Symposium on Knowledge Acquisition and Modeling.

[31]  Ning Huang,et al.  Structure-Based Ontology Evaluation , 2006, ICEBE.

[32]  Ricardo de Almeida Falbo,et al.  Ontologies in Software Testing: A Systematic Literature Review , 2013, ONTOBRAS.

[33]  Marko Grobelnik,et al.  A SURVEY OF ONTOLOGY EVALUATION TECHNIQUES , 2005 .

[34]  Karla Okada,et al.  SwTO I ( Software Test Ontology Integrated ) and its Application in Linux Test , 2009 .

[35]  Aldo Gangemi,et al.  Is there beauty in ontologies? , 2011, Appl. Ontology.

[36]  Dieter Pfoser Indexing the Trajectories of Moving Objects , 2002 .

[37]  Krzysztof Wnuk,et al.  Knowledge Management in Software Testing: A Systematic Snowball Literature Review , 2018, e Informatica Softw. Eng. J..

[38]  Giancarlo Guizzardi,et al.  On Ontology, ontologies, Conceptualizations, Modeling Languages, and (Meta)Models , 2007, DB&IS.

[39]  Ricardo de Almeida Falbo,et al.  SABiO: Systematic Approach for Building Ontologies , 2014, ONTO.COM/ODISE@FOIS.

[40]  Jean-Paul A. Barthès,et al.  Knowledge Management , 1994, Encyclopedia of Database Systems.

[41]  Fernanda Campos,et al.  Regression Tests Provenance Data in the Continuous Software Engineering Context , 2017, SAST.

[42]  Norbert Luttenberger,et al.  From UML to OWL 2 , 2011, KTW.

[43]  Chin-Feng Fan,et al.  Systematic scenario test case generation for nuclear safety systems , 2013, Inf. Softw. Technol..

[44]  Ricardo de Almeida Falbo,et al.  An Infrastructure for Managing Semantic Documents , 2010, 2010 14th IEEE International Enterprise Distributed Object Computing Conference Workshops.

[45]  Ji Wu,et al.  Investigation of Knowledge Management Methods in Software Testing Process , 2009, 2009 International Conference on Information Technology and Computer Science.

[46]  Tore Dybå,et al.  Empirical studies of agile software development: A systematic review , 2008, Inf. Softw. Technol..

[47]  Sandro Morasca,et al.  Supporting the semi-automatic semantic annotation of web services: A systematic literature review , 2015, Inf. Softw. Technol..

[48]  Phil McMinn,et al.  Search-Based Test Input Generation for String Data Types Using the Results of Web Queries , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[49]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[50]  M. Lindvall,et al.  Knowledge management in software engineering , 2002, IEEE Software.

[51]  Ahmad Kayed,et al.  Towards an Ontology for Software Product Quality Attributes , 2009, 2009 Fourth International Conference on Internet and Web Applications and Services.

[52]  María José Escalona Cuaresma,et al.  A Model-Driven approach for functional test case generation , 2015, J. Syst. Softw..

[53]  Kai Petersen,et al.  SERP-test: a taxonomy for supporting industry–academia communication , 2017, Software Quality Journal.

[54]  Han Li,et al.  Using ontology to generate test cases for GUI testing , 2011, Int. J. Comput. Appl. Technol..

[55]  R. Studer,et al.  Semantic Web Technologies: Trends and Research in Ontology-based Systems , 2006 .

[56]  Weichang Du,et al.  An Ontology-based Software Test Generation Framework , 2010, SEKE.

[57]  Stefan Biffl,et al.  Simulation of workflow scheduling strategies using the MAST test management system , 2008, 2008 10th International Conference on Control, Automation, Robotics and Vision.

[58]  Kristof Meixner,et al.  Securing the testing process for industrial automation software , 2019, Comput. Secur..

[59]  Weiqin Tong,et al.  An application of ontology to test case reuse , 2011, 2011 International Conference on Mechatronic Science, Electric Engineering and Computer (MEC).

[60]  Anna Perini,et al.  Ontology-based test generation for multiagent systems , 2008, AAMAS.

[61]  Sussy Bayona Oré,et al.  Critical success factors taxonomy for software process deployment , 2012, Software Quality Journal.

[62]  Aldo Gangemi,et al.  Unit Tests for Ontologies , 2006, OTM Workshops.

[63]  Ricardo de Almeida Falbo,et al.  ROoST: Reference Ontology on Software Testing , 2017, Appl. Ontology.

[64]  Jean Vanderdonckt,et al.  A Unifying Reference Framework for multi-target user interfaces , 2003, Interact. Comput..

[65]  Kai Petersen,et al.  Guidelines for conducting systematic mapping studies in software engineering: An update , 2015, Inf. Softw. Technol..

[66]  Michael Uschold,et al.  Ontologies: principles, methods and applications , 1996, The Knowledge Engineering Review.

[67]  Asunción Gómez-Pérez,et al.  Ontology Evaluation , 2004, Handbook on Ontologies.

[68]  Akshi Kumar,et al.  Ontologies for Software Engineering: Past, Present and Future , 2016 .

[69]  Vijayan Sugumaran,et al.  A semiotic metrics suite for assessing the quality of ontologies , 2005, Data Knowl. Eng..

[70]  Ralf Tönjes,et al.  Automated Testing of Context-Aware Applications , 2015, 2015 IEEE 82nd Vehicular Technology Conference (VTC2015-Fall).

[71]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[72]  Tim Berners-Lee,et al.  Linked Data - The Story So Far , 2009, Int. J. Semantic Web Inf. Syst..

[73]  Panagiotis Manolios,et al.  Automating requirements analysis and test case generation , 2019, Requirements Engineering.

[74]  Torgeir Dingsøyr,et al.  Knowledge management in software engineering: A systematic review of studied concepts, findings and research methods used , 2008, Inf. Softw. Technol..

[75]  Jing Tian,et al.  An Ontology-based Knowledge Management System for Software Testing , 2017, SEKE.

[76]  Francisco J. García-Peñalvo,et al.  SABUMO-dTest: Design and evaluation of an intelligent collaborative distributed testing framework , 2014, Comput. Sci. Inf. Syst..

[77]  José Ramón Hilera,et al.  Combining Multiple Web Accessibility Evaluation Reports using Semantic Web Technologies , 2017, ISD.

[78]  Raghu Singh,et al.  International Standard ISO/IEC 12207 Software Life Cycle Processes , 1996, Softw. Process. Improv. Pract..

[79]  Standard Glossary of Terms used in Software Testing Version 3 . 1 Foundation Terms , .

[80]  Atif M. Memon,et al.  GUI ripping: reverse engineering of graphical user interfaces for testing , 2003, 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings..

[81]  Standard Glossary of Software Engineering Terminology , 1990 .

[82]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[83]  Franz Wotawa,et al.  On the Industrial Application of Combinatorial Testing for Autonomous Driving Functions , 2019, 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[84]  N. F. Noy,et al.  Ontology Development 101: A Guide to Creating Your First Ontology , 2001 .

[85]  Michael Gruninger,et al.  Methodology for the Design and Evaluation of Ontologies , 1995, IJCAI 1995.

[86]  R. Li,et al.  The Use of Ontology in Case Based Reasoning for Reusable Test Case Generation , 2015 .

[87]  Mark Harman,et al.  Automatically generating realistic test input from web services , 2011, Proceedings of 2011 IEEE 6th International Symposium on Service Oriented System (SOSE).

[88]  Iman Keivanloo,et al.  Software trustworthiness 2.0 - A semantic web enabled global source code analysis approach , 2014, J. Syst. Softw..

[89]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[90]  Stefan Biffl,et al.  Ontology-Based Test Case Generation For Simulating Complex Production Automation Systems , 2010, SEKE.

[91]  Javier Andrade Garda,et al.  An architectural model for software testing lesson learned systems , 2013, Inf. Softw. Technol..

[92]  Valeriy Vyatkin,et al.  Requirements-Aided Automatic Test Case Generation for Industrial Cyber-physical Systems , 2015, 2015 20th International Conference on Engineering of Complex Computer Systems (ICECCS).

[93]  Hong Zhu,et al.  A multi-agent software engineering environment for testing Web-based applications , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[94]  Giancarlo Guizzardi,et al.  Towards an Ontology of Software Defects, Errors and Failures , 2018, ER.

[95]  P. Kalaivani,et al.  Development Of Ontology-Based Intelligent System For Software Testing , 2013, ArXiv.

[96]  Ricardo de Almeida Falbo,et al.  Using Ontology Patterns for Building a Reference Software Testing Ontology , 2013, 2013 17th IEEE International Enterprise Distributed Object Computing Conference Workshops.

[97]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[98]  Christopher Menzel,et al.  Reference Ontologies - Application Ontologies: Either/Or or Both/And? , 2003, KI Workshop on Reference Ontologies and Application Ontologies.

[99]  Marco Winckler,et al.  Ensuring the Consistency Between User Requirements and Graphical User Interfaces: A Behavior-Based Automated Approach , 2019, ICCSA.

[100]  Guntis Arnicans,et al.  Semi-Automatic Generation of a Software Testing Lightweight Ontology from a Glossary Based on the ONTO6 Methodology , 2012, DB&IS.

[101]  Marco Winckler,et al.  A Behavior-Based Ontology for Supporting Automated Assessment of Interactive Systems , 2019, 2017 IEEE 11th International Conference on Semantic Computing (ICSC).

[102]  Mick Kerrigan,et al.  An Ontology Model to Support the Automated Evaluation of Software , 2010, SEKE.

[103]  William C. Chu,et al.  An Ontology-Based Approach for GUI Testing , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[104]  Renata Vieira,et al.  An Ontology for Guiding Performance Testing , 2014, 2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT).

[105]  Peng Liang,et al.  Knowledge-based approaches in software documentation: A systematic literature review , 2014, Inf. Softw. Technol..

[106]  Diego Calvanese,et al.  Reasoning on UML class diagrams , 2005, Artif. Intell..

[107]  Anna Perini,et al.  Experimental Evaluation of Ontology-Based Test Generation for Multi-agent Systems , 2009, AOSE.

[108]  Mikael Lindvall,et al.  Practical Implications of Traceability , 1996, Softw. Pract. Exp..

[109]  Birgit Vogel-Heuser,et al.  Keeping requirements and test cases consistent: Towards an ontology-based approach , 2014, 2014 12th IEEE International Conference on Industrial Informatics (INDIN).

[110]  Hong Zhu,et al.  Ontology for Service Oriented Testing of Web Services , 2008, 2008 IEEE International Symposium on Service-Oriented System Engineering.

[111]  Hong Zhu,et al.  A Framework for Service-Oriented Testing of Web Services , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[112]  Thomas R. Gruber,et al.  A translation approach to portable ontology specifications , 1993, Knowl. Acquis..

[113]  Iván García-Magariño,et al.  Definition of Process Models for Agent-Based Development , 2008, AOSE.

[114]  Aining Yin,et al.  Ontology development for unified traditional Chinese medical language system , 2004, Artif. Intell. Medicine.

[115]  Sakorn Mekruksavanich Ontology-Assisted Structural Design Flaw Detection of Object-Oriented Software , 2017 .

[116]  Hong Zhu,et al.  Collaborative Testing of Web Services , 2012, IEEE Transactions on Services Computing.

[117]  James Cheney,et al.  PROV-O: The PROV ontology:W3C recommendation 30 April 2013 , 2013 .

[118]  Ana Regina Cavalcanti da Rocha,et al.  A Systematic Approach for Building Ontologies , 1998, IBERAMIA.

[119]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[120]  Vladimir Tarasov,et al.  Application of Inference Rules to a Software Requirements Ontology to Generate Software Test Cases , 2016, OWLED.

[121]  Ricardo Colomo Palacios,et al.  SABUMO: Towards a collaborative and semantic framework for knowledge sharing , 2012, Expert Syst. Appl..

[122]  Ellis E. Eghan,et al.  Recovering Semantic Traceability Links between APIs and Security Vulnerabilities: An Ontological Modeling Approach , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[123]  Jéssyka Vilela,et al.  Applications of ontologies in requirements engineering: a systematic review of the literature , 2015, Requirements Engineering.

[124]  Katarina Britz,et al.  Scenario Testing Using Formal Ontologies , 2014, ONTO.COM/ODISE@FOIS.

[125]  Zoltán Szatmári,et al.  Ontology-based Test Data Generation using Metaheuristics , 2011, ICINCO.

[126]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[127]  Waralak V. Siricharoen Ontologies and Software Engineering , 2007, International Conference on Computational Science.

[128]  Weichang Du,et al.  Knowledge-based Software Test Generation , 2009, SEKE.

[129]  Fabiano Cutigi Ferrari,et al.  An aspect-oriented reference architecture for Software Engineering Environments , 2011, J. Syst. Softw..

[130]  Elisa Yumi Nakagawa,et al.  Towards the Establishment of an Ontology of Software Testing , 2006, SEKE.

[131]  Jing Tian,et al.  A Survey on Knowledge Management in Software Engineering , 2015, 2015 IEEE International Conference on Software Quality, Reliability and Security - Companion.

[132]  Rex Black,et al.  Foundations of Software Testing ISTQB Certification , 2006 .

[133]  Anna Perini,et al.  eCAT: a tool for automating test cases generation and execution in testing multi-agent systems , 2008, AAMAS.

[134]  Sandro Morasca,et al.  systematic review on the functional testing of semantic web services , 2013 .

[135]  Débora Maria Barroso Paiva,et al.  ONTO-ResAsset Development: An Ontology for Reusable Assets Specification and Management , 2014, SEKE.

[136]  Richard E. Fairley,et al.  Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0 , 2014 .

[137]  Enrico Vicario,et al.  Using Ontologies in the Integration of Structural, Functional, and Process Perspectives in the Development of Safety Critical Systems , 2013, Ada-Europe.

[138]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[139]  M. Merdan,et al.  Performance evaluation of workflow scheduling strategies considering transportation times and conveyor failures , 2008, 2008 IEEE International Conference on Industrial Engineering and Engineering Management.

[140]  Adam Pease,et al.  Towards a standard upper ontology , 2001, FOIS.

[141]  Mario Jino,et al.  A Security Testing Process Supported by an Ontology Environment: A Conceptual Proposal , 2018, 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA).

[142]  Guntis Arnicans,et al.  Transformation of the Software Testing Glossary into a Browsable Concept Map , 2015 .

[143]  Hong Zhu,et al.  Developing Software Testing Ontology in UML for a Software Growth Environment of Web-Based Applications , 2005 .

[144]  Abdul Rauf,et al.  Validation of ontology based test case generation for graphical user interface , 2012, 2012 15th International Multitopic Conference (INMIC).

[145]  Sandeep Kumar,et al.  An Ontology-Based Approach for Test Case Reuse , 2015 .

[146]  Yao Zhang,et al.  Semantic-Based Test Oracles , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference.

[147]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[148]  Samuel J. Collins UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION OFFICE OF NUCLEAR REACTOR REGULATION , 1997 .