Automation and intelligent scheduling of distributed system functional testing

This paper presents the approach to functional test automation of services (black-box testing) and service architectures (grey-box testing) that has been developed within the MIDAS project and is accessible on the MIDAS SaaS. In particular, the algorithms and techniques adopted for addressing input and oracle generation, dynamic scheduling, and session planning issues supporting service functional test automation are illustrated. More specifically, the paper details: (i) the test input generation based on formal methods and temporal logic specifications, (ii) the test oracle generation based on service formal specifications, (iii) the dynamic scheduling of test cases based on probabilistic graphical reasoning, and (iv) the reactive, evidence-based planning of test sessions with on-the-fly generation of new test cases. Finally, the utilisation of the MIDAS prototype for the functional test of operational services and service architectures in the healthcare industry is reported and assessed. A planned evolution of the technology deals with the testing and troubleshooting of distributed systems that integrate connected objects.

[1]  Raymond A. Paul,et al.  Adaptive testing, oracle generation, and test case ranking for Web services , 2005, 29th Annual International Computer Software and Applications Conference (COMPSAC'05).

[2]  Rance Cleaveland,et al.  Using formal specifications to support testing , 2009, CSUR.

[3]  Cheng Zhang,et al.  Automated Test Oracles: A Survey , 2015, Adv. Comput..

[4]  Simon Parsons,et al.  Probabilistic Graphical Models: Principles and Techniques by Daphne Koller and Nir Friedman, MIT Press, 1231 pp., $95.00, ISBN 0-262-01319-3 , 2011, The Knowledge Engineering Review.

[5]  Marko Becker,et al.  Service Oriented Architecture Concepts Technology And Design , 2016 .

[6]  Mark Harman,et al.  Testing and verification in service‐oriented architecture: a survey , 2013, Softw. Test. Verification Reliab..

[7]  Philip Samuel,et al.  Web services regression test case prioritization , 2010, 2010 International Conference on Computer Information Systems and Industrial Management Applications (CISIM).

[8]  Marijn Haverbeke,et al.  Eloquent JavaScript: A Modern Introduction to Programming , 2011 .

[9]  Yen-Ting Lee,et al.  Automatic SaaS test cases generation based on SOA in the cloud service , 2012, 4th IEEE International Conference on Cloud Computing Technology and Science Proceedings.

[10]  Anders L. Madsen,et al.  LAZY Propagation: A Junction Tree Inference Algorithm Based on Lazy Evaluation , 1999, Artif. Intell..

[11]  Franz Wotawa,et al.  Fifty Shades of Grey in SOA Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[12]  Chris Newcombe.  Why Amazon Chose TLA + , 2014, ABZ.

[13]  Paolo Tonella,et al.  Change Sensitivity Based Prioritization for Audit Testing of Webservice Compositions , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[14]  Eda Marchetti,et al.  WS-TAXI: A WSDL-based Testing Tool for Web Services , 2009, 2009 International Conference on Software Testing Verification and Validation.

[15]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[16]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[17]  Leslie Lamport,et al.  Checking Cache-Coherence Protocols with TLA+ , 2003, Formal Methods Syst. Des..

[18]  Maria Grazia Fugini,et al.  WS-DIAMOND an approach to Web services diagnosability, Monitoring and diagnosis , 2007 .

[19]  Raymond Reiter,et al.  A Theory of Diagnosis from First Principles , 1986, Artif. Intell..

[20]  Mariëlle Stoelinga,et al.  Talking quiescence: a rigorous theory that supports parallel composition, action hiding and determinisation , 2012, MBT.

[21]  Leonidas Lampropoulos,et al.  Automatic WSDL-guided Test Case Generation for PropEr Testing of Web Services , 2012, WWV.

[22]  Arkady B. Zaslavsky,et al.  Sensing as a service model for smart cities supported by Internet of Things , 2013, Trans. Emerg. Telecommun. Technol..

[23]  Aleksandr Yakovlevich Khinchin,et al.  Mathematical foundations of information theory , 1959 .

[24]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[25]  Niels Lohmann,et al.  Realizability is controllability , 2009, ZEUS.

[26]  Cesare Pautasso,et al.  REST: From Research to Practice , 2011 .

[27]  Hongbing Wang,et al.  Describing and Verifying Web Service Composition Using TLA Reasoning , 2010, 2010 IEEE International Conference on Services Computing.

[28]  Eric Newcomer,et al.  Understanding Web Services: XML, WSDL, SOAP, and UDDI , 2002 .

[29]  Edmund M. Clarke,et al.  Model Checking and the State Explosion Problem , 2011, LASER Summer School.

[30]  Philippe Schnoebelen,et al.  The Complexity of Temporal Logic Model Checking , 2002, Advances in Modal Logic.

[31]  Frank P. A. Coolen,et al.  Managing the uncertainties of software testing: a Bayesian approach , 2001 .

[32]  Tien-Dung Cao,et al.  Online Testing Framework for Web Services , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[33]  Eric. Newcomer,et al.  Understanding SOA with Web Services , 2004 .

[34]  Kathrin Kaschner,et al.  Automatic Test Case Generation for Interacting Services , 2009, ICSOC Workshops.

[35]  Michael Goldstein,et al.  Bayesian Graphical Models for Software Testing , 2002, IEEE Trans. Software Eng..

[36]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[37]  D Conforti,et al.  EHMTI-0172. “Calabria cephalalgic network”: innovative services and systems for the integrated clinical management of headache patients , 2014, The Journal of Headache and Pain.

[38]  Paolo Arcaini,et al.  Optimizing the automatic test generation by SAT and SMT solving for Boolean expressions , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[39]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[40]  Fatiha Zaïdi,et al.  Checking the Behavioral Conformance of Web Services with Symbolic Testing and an SMT Solver , 2011, TAP@TOOLS.

[41]  Michel Wermelinger,et al.  Fundamental Approaches to Software Engineering: 7th International Conference Proceedings , 2004 .

[42]  Schahram Dustdar,et al.  GENESIS - A Framework for Automatic Generation and Steering of Testbeds of ComplexWeb Services , 2008, 13th IEEE International Conference on Engineering of Complex Computer Systems (iceccs 2008).

[43]  Franz Wotawa,et al.  Functional SOA testing based on constraints , 2013, 2013 8th International Workshop on Automation of Software Test (AST).

[44]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[45]  M. A. Barcelona,et al.  Practical experiences in the usage of MIDAS in the logistics domain , 2016, International Journal on Software Tools for Technology Transfer.

[46]  Ariele Maesano,et al.  Bayesian dynamic scheduling for service composition testing , 2015 .

[47]  Leslie Lamport,et al.  Specifying Systems: The TLA+ Language and Tools for Hardware and Software Engineers [Book Review] , 2002, Computer.

[48]  Eric. Newcomer,et al.  Understanding SOA with Web Services (Independent Technology Guides) , 2004 .

[49]  Carsten Bormann,et al.  The Constrained Application Protocol (CoAP) , 2014, RFC.

[50]  Thomas Erl,et al.  Service-Oriented Architecture: Concepts, Technology, and Design , 2005 .

[51]  Ladan Tahvildari,et al.  A Prioritization Approach for Software Test Cases Based on Bayesian Networks , 2007, FASE.

[52]  Claus Skaanning,et al.  Printer Troubleshooting Using Bayesian Networks , 2000, IEA/AIE.

[53]  Rina Dechter.  Bucket Elimination: a Unifying Framework for Processing Hard and Soft Constraints , 2004, Constraints.

[54]  Akbar Siami Namin,et al.  Bayesian reasoning for software testing , 2010, FoSER '10.

[55]  Liisa Salparanta.  From research to practice in Finland , 2011 .

[56]  Baowen Xu,et al.  Test Case Prioritization for Web Service Regression Testing , 2010, 2010 Fifth IEEE International Symposium on Service Oriented System Engineering.

[57]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[58]  A. Askarunisa,et al.  Black box test case prioritization techniques for semantic based composite web services using OWL-S , 2011, 2011 International Conference on Recent Trends in Information Technology (ICRTIT).

[59]  Brian C. Williams,et al.  Diagnosing Multiple Faults , 1987, Artif. Intell..

[60]  Philip Mayer,et al.  Towards a BPEL unit testing framework , 2006, TAV-WEB '06.

[61]  Sam Newman,et al.  Building microservices - designing fine-grained systems, 1st Edition , 2015 .

[62]  T. H. Tse,et al.  XML-manipulating test case prioritization for XML-manipulating services , 2011, J. Syst. Softw..

[63]  Leslie Lamport,et al.  The PlusCal Algorithm Language , 2009, ICTAC.

[64]  Ebrahim Shamsoddin-Motlagh.  A survey of service oriented architecture systems testing , 2012, ArXiv.

[65]  Jirí Vomlel,et al.  The SACSO methodology for troubleshooting complex systems , 2001, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[66]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .