Test’n’Mo: a collaborative platform for human testers and intelligent monitoring agents

Many software bugs have disruptive consequences, both in financial terms and in loss of life. Software Testing is one widely used approach to detect software bugs and ensure software quality but the testing activity, conducted either manually or using testing frameworks, is repetitive and expensive. Runtime Monitoring, differently from Software Testing, does not require test cases to be designed and executed and – once the property to be monitored has been specified – it does not rely on human beings performing any further actions, unless a violation is detected. However the property to be monitored, that must feed the monitor along with the trace or stream of observed events, may be very hard to identify and specify. In this extended abstract we present the Test'n'Mo vision which goes in the direction of exploiting Artificial intelligence and Machine Learning as enabling techniques for a hybrid platform for Software Testing and Runtime Monitoring. In Test'n'Mo, human testers and software agents of different kinds – 'Learning Agents' and 'Runtime Monitoring and Testing Agents' – collaborate to achieve their common testing goal. Although Test'n'Mo is meant to address User Interface testing of web/mobile apps, the Test'n'Mo approach may be adapted to other software testing activities.

[1]  Frank Dignum,et al.  An Appraisal Transition System for Event-driven Emotions in Agent-based Player Experience Testing , 2021, EMAS@AAMAS.

[2]  Angelo Ferrando,et al.  RML: Theory and practice of a domain specific language for runtime verification , 2021, Sci. Comput. Program..

[3]  Filippo Ricca,et al.  AI-based Test Automation: A Grey Literature Analysis , 2021, 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[4]  Eduard Paul Enoiu,et al.  Agent-Based Software Testing: A Definition and Systematic Mapping Study , 2020, 2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C).

[5]  Boni García,et al.  A Survey of the Selenium Ecosystem , 2020 .

[6]  Lefteris Moussiades,et al.  An Overview of Chatbot Technology , 2020, AIAI.

[7]  Ralf Lämmel,et al.  WES: Agent-based User Interaction Simulation on Real Infrastructure , 2020, ICSE.

[8]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[9]  Eduard Paul Enoiu,et al.  Test Agents: The Next Generation of Test Cases , 2019, 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[10]  Kenneth O. Stanley,et al.  Go-Explore: a New Approach for Hard-Exploration Problems , 2019, ArXiv.

[11]  Dejan Nickovic,et al.  A survey of challenges for runtime verification from advanced application domains (beyond software) , 2018, Formal Methods in System Design.

[12]  Paolo Tonella,et al.  Search Based Path and Input Data Generation for Web Application Testing , 2017, SSBSE.

[13]  Vladimiro Sassone,et al.  Decentralised Runtime Monitoring for Access Control Systems in Cloud Federations , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[14]  Viviana Mascardi,et al.  Can My Test Case Run on Your Test Plant? A Logic-Based Compliance Check and Its Evaluation on Real Data , 2017, RuleML+RR.

[15]  Morten Mossige,et al.  Reinforcement learning for automatic test case prioritization and selection in continuous integration , 2017, ISSTA.

[16]  Vahid Garousi,et al.  Developing, Verifying, and Maintaining High-Quality Automated Test Scripts , 2016, IEEE Software.

[17]  Anita Ganpati,et al.  Identification of attributes for test case reusability in regression test selection techniques , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[18]  Doo-Hwan Bae,et al.  A runtime verification framework for dynamically adaptive multi-agent systems , 2016, 2016 International Conference on Big Data and Smart Computing (BigComp).

[19]  Tanja E. J. Vos,et al.  TESTAR: Tool Support for Test Automation at the User Interface Level , 2015, Int. J. Inf. Syst. Model. Des..

[20]  Angelo Ferrando,et al.  Global Protocols as First Class Entities for Self-Adaptive Agents , 2015, AAMAS.

[21]  A. Deursen,et al.  Crawling Ajax-Based Web Applications through Dynamic Analysis of User Interface State Changes , 2012, TWEB.

[22]  Li Ying,et al.  Test Case Reusability Metrics Model , 2010, 2010 2nd International Conference on Computer Technology and Development.

[23]  Wojciech Penczek,et al.  Runtime Monitoring of Contract Regulated Web Services , 2010, Fundam. Informaticae.

[24]  Hao Cai,et al.  A comparison of three agent-oriented software development methodologies: MaSE, Gaia, and Tropos , 2009, 2009 IEEE Youth Conference on Information, Computing and Telecommunication.

[25]  Andrea Omicini,et al.  Artifacts in the A&A meta-model for multi-agent systems , 2008, Autonomous Agents and Multi-Agent Systems.

[26]  Michael Wooldridge,et al.  Programming Multi-Agent Systems in AgentSpeak using Jason (Wiley Series in Agent Technology) , 2007 .

[27]  Agostino Poggi,et al.  Developing Multi-agent Systems with JADE , 2007, ATAL.

[28]  Wei-Tek Tsai,et al.  A multi-agent based framework for collaborative testing on Web services , 2006, The Fourth IEEE Workshop on Software Technologies for Future Embedded and Ubiquitous Systems, and the Second International Workshop on Collaborative Computing, Integration, and Assurance (SEUS-WCCIA'06).

[29]  Kamalendu Pal,et al.  Framework for Reusable Test Case Generation in Software Systems Testing , 2020, Software Engineering for Agile Application Development.

[30]  Filippo Ricca,et al.  Chapter Three - Three Open Problems in the Context of E2E Web Testing and a Vision: NEONATE , 2019, Adv. Comput..

[31]  Angelo Ferrando,et al.  Comparing Trace Expressions and Linear Temporal Logic for Runtime Verification , 2016, Theory and Practice of Formal Methods.

[32]  Paolo Tonella,et al.  Approaches and Tools for Automated End-to-End Web Testing , 2016, Adv. Comput..

[33]  Gail E. Kaiser,et al.  Properties of Machine Learning Applications for Use in Metamorphic Testing , 2008, SEKE.

[34]  Ehud Shapiro,et al.  Inductive Inference of Theories from Facts , 1991, Computational Logic - Essays in Honor of Alan Robinson.

[35]  G. Plotkin Automatic Methods of Inductive Inference , 1972 .