Agent-Based Software Testing: A Definition and Systematic Mapping Study

The emergence of new technologies in software testing has increased the automation and flexibility of the testing process. In this context, the adoption of agents in software testing remains an active research area in which various agent methodologies, architectures, and tools are employed to improve different test problems. Even though research that investigates agents in software testing has been growing, these agent-based techniques should be considered in a broader perspective. In order to provide a comprehensive overview of this research area, which we define as agent-based software testing (ABST), a systematic mapping study has been conducted. This mapping study aims to identify the topics studied within ABST, as well as examine the adopted research methodologies, identify the gaps in the current research and point to directions for future ABST research. Our results suggest that there is an interest in ABST after 1999 that resulted in the development of solutions using reactive, BDI, deliberative and cooperate agent architectures for software testing. In addition, most of the ABST approaches are designed using the JADE framework, have targeted the Java programming language, and are used at system-level testing for functional, non-functional and white-box testing. In regards to regression testing, our results indicate a research gap that could be addressed in future studies.

[1]  Guoshun Chen,et al.  A kind of software fault diagnosing framework based on Multi-Agent , 2012, 2012 International Conference on Quality, Reliability, Risk, Maintenance, and Safety Engineering.

[2]  Liang Liu,et al.  Agent-based automated compatibility software test for NLSF , 2003, Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.03EX693).

[3]  Robert G. Reynolds,et al.  Knowledge-based software testing agent using evolutionary learning with cultural algorithms , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[4]  Kangsoo Kim,et al.  A Systematic Survey of 15 Years of User Studies Published in the Intelligent Virtual Agents Conference , 2018, IVA.

[5]  M. L. Valarmathi,et al.  Multi Agent Based Framework for Structural and Model Based Test Case Generation , 2012 .

[6]  J. Friedenberg,et al.  Cognitive Science: An Introduction to the Study of Mind , 2005 .

[7]  Rayner Alfred,et al.  Agent Architecture: An Overviews , 2014 .

[8]  Samia Nefti-Meziani,et al.  Autonomous agents for Testing and Verification of Softwares after Deployment over Network , 2011, 2011 World Congress on Internet Security (WorldCIS-2011).

[9]  Yunzhan Gong,et al.  Design of intelligent agents for collaborative testing of service-based systems , 2011, AST '11.

[10]  Hong Zhu.  Cooperative agent approach to quality assurance and testing Web software , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[11]  Jia Zhang,et al.  A Mobile Agent-Supported Web Services Testing Platform , 2008, 2008 IEEE/IFIP International Conference on Embedded and Ubiquitous Computing.

[12]  Michael Winikoff,et al.  BDI agent testability revisited , 2017, Autonomous Agents and Multi-Agent Systems.

[13]  Miquel Sànchez-Marrè,et al.  A review on multi-agent platforms and environmental decision support systems simulation tools , 2006 .

[14]  Julien Provost,et al.  Framework for Flexible, Adaptive Support of Test Management by Means of Software Agents , 2019, IEEE Robotics and Automation Letters.

[15]  Katia P. Sycara,et al.  Intelligent Adaptive Information Agents , 1997, Journal of Intelligent Information Systems.

[16]  N. Ramaraj,et al.  Enhancing the Efficiency of Regression Testing through Intelligent Agents , 2007, International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007).

[17]  Luiz Fernando Capretz,et al.  A Multi-Agent Framework for Testing Distributed Systems , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[18]  Xiaofeng Wang,et al.  Research on automated testing framework based on ontology and multi-agent , 2010, 2010 Third International Symposium on Knowledge Acquisition and Modeling.

[19]  Alexander Egyed,et al.  A first systematic mapping study on combinatorial interaction testing for software product lines , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[20]  Barton P. Miller,et al.  Diagnosing Distributed Systems with Self-propelled Instrumentation , 2008, Middleware.

[21]  Nasser Jazdi,et al.  Prioritization of Test Cases Using Software Agents and Fuzzy Logic , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[22]  Eduard Paul Enoiu,et al.  Test Agents: The Next Generation of Test Cases , 2019, 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[23]  J. P. Lanslots,et al.  Using intelligent agents for distributed measurements , 2003, IEEE International Symposium on Intelligent Signal Processing, 2003.

[24]  Lakhmi C. Jain,et al.  Innovations in Multi-Agent Systems and Applications - 1 , 2010 .

[25]  Alfredo Benso,et al.  Agent based dbist dbisr and its web wireless management , 2003, International Test Conference, 2003. Proceedings. ITC 2003..

[26]  Kai Petersen,et al.  Mapping software testing practice with software testing research — SERP-test taxonomy , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[27]  Roel Wieringa,et al.  Requirements engineering paper classification and evaluation criteria: a proposal and a discussion , 2005, Requirements Engineering.

[28]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[29]  Anna Perini,et al.  eCAT: a tool for automating test cases generation and execution in testing multi-agent systems , 2008, AAMAS.

[30]  Wang Yinglong,et al.  A IPv6 Network Performance Test System using Multi-Agent , 2007, 2007 8th International Conference on Electronic Measurement and Instruments.

[31]  Bin Chen,et al.  Design of BDI Agent for Adaptive Performance Testing of Web Services , 2010, 2010 10th International Conference on Quality Software.

[32]  Nanjangud C. Narendra,et al.  Large scale testing of pervasive computing systems using multi-agent simulation , 2005, Third International Workshop on Intelligent Solutions in Embedded Systems, 2005..

[33]  Michael Winikoff.  Future directions for agent-based software engineering , 2009, Int. J. Agent Oriented Softw. Eng..

[34]  L. Miclea,et al.  Agent-based testing and repair of heterogeneous distributed systems , 2008, 2008 IEEE International Conference on Automation, Quality and Testing, Robotics.

[35]  V. Mohan,et al.  IntelligenTester - Software Test Sequence Optimization Using Graph Based Intelligent Search Agent , 2007, International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007).

[36]  David Chenho Kung,et al.  An agent-based framework for testing Web applications , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[37]  Dalton Serey Guerrero,et al.  Distributed test agents: a pattern for the development of automatic system tests for distributed applications , 2012, SugarLoafPLoP '12.

[38]  Theodore D. Hellmann,et al.  Agile Testing: A Systematic Mapping across Three Conferences: Understanding Agile Testing in the XP/Agile Universe, Agile, and XP Conferences , 2013, 2013 Agile Conference.

[39]  Hong Zhu,et al.  A multi-agent software engineering environment for testing Web-based applications , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[40]  G. V. Uma,et al.  A new approach in development of distributed framework for automated software testing using agents , 2006, Knowl. Based Syst..

[41]  Pearl Brereton,et al.  A Systematic Mapping Study of Empirical Studies on Software Cloud Testing Methods , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C).

[42]  Jingfan Tang.  Towards Automation in Software Test Life Cycle Based on Multi-Agent , 2010, 2010 International Conference on Computational Intelligence and Software Engineering.

[43]  C. Malz,et al.  Agent-based test management for software system test , 2010, 2010 IEEE International Conference on Automation, Quality and Testing, Robotics (AQTR).

[44]  Hüsnü Yenigün,et al.  Testing Software and Systems , 2015, Lecture Notes in Computer Science.

[45]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[46]  G. V. Uma,et al.  Multi-agent Based Integrated Framework for Intra-class Testing of Object-Oriented Software , 2003, ISCIS.

[47]  Antonio Ruiz-Cortés,et al.  Replication of Studies in Empirical Software Engineering: A Systematic Mapping Study, From 2013 to 2018 , 2020, IEEE Access.

[48]  Fernando Buarque de Lima Neto,et al.  Improving black box testing by using neuro-fuzzy classifiers and multi-agent systems , 2010, 2010 10th International Conference on Hybrid Intelligent Systems.

[49]  Nicholas R. Jennings,et al.  Intelligent agents: theory and practice , 1995, The Knowledge Engineering Review.

[50]  Ali Yazici,et al.  Systematic Mapping on Quality in Web Application Testing , 2019, 2019 1st International Informatics and Software Engineering Conference (UBMYK).

[51]  Nicos Malevris,et al.  Employing Agents Towards Database Applications Testing , 2007, 19th IEEE International Conference on Tools with Artificial Intelligence(ICTAI 2007).

[52]  Michael Wooldridge,et al.  Agent-based software engineering , 1997, IEE Proc. Softw. Eng..

[53]  Nina Yevtushenko,et al.  Testing software and systems - 29th IFIP WG 6.1 International Conference (ICTSS 2017) , 2017 .

[54]  John Aycock,et al.  A Multi-Agent Approach to Testing Anti-Spam Software , 2009, 2009 20th International Workshop on Database and Expert Systems Application.

[55]  Carlos José Pereira de Lucena,et al.  JAAF+T: a framework to implement self-adaptive agents that apply self-test , 2010, SAC '10.

[56]  Yuqing Lan,et al.  Automatic Test Task Allocation in Agent-Based Distributed Automated Testing Framework , 2009, 2009 International Conference on Computational Intelligence and Software Engineering.

[57]  Riccardo Scandariato,et al.  Current and Future Bots in Software Development , 2019, 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE).

[58]  Michael Wooldridge,et al.  Intelligent agents: theory and practice The Knowledge Engineering Review , 1995 .

[59]  Y. Shoham.  Introduction to Multi-Agent Systems , 2002 .

[60]  Rosario Girardi,et al.  A Survey on Software Agent Architectures , 2013, IEEE Intell. Informatics Bull..

[61]  Peter Göhner,et al.  Agent-Based Test Case Prioritization , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[62]  D. Jeya Mala,et al.  IntelligenTester - Test Sequence Optimization Framework using Multi-Agents , 2008, J. Comput..

[63]  Stefan Karlsson.  Exploratory test agents for stateful software systems , 2019, ESEC/SIGSOFT FSE.

[64]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[65]  Zaidi Sahnoun,et al.  MAEST :Multi-Agent Environment for Software Testing , 2007 .

[66]  John C. Grundy,et al.  Deployed software component testing using dynamic validation agents , 2005, J. Syst. Softw..