A Template-Based Approach to Describing Metamorphic Relations

Metamorphic testing enables the generation of test cases in the absence of an oracle by exploiting relations among different executions of the program under test, called metamorphic relations. In a recent survey, we observed a great variability in the way metamorphic relations are described, typically in an informal manner using natural language. We noticed that the lack of a standard mechanism to describe metamorphic relations often makes them hard to read and understand, which hinders the widespread adoption of the technique. In this paper, we propose a template-based approach for the description of metamorphic relations. The proposed template aims to ease communication among practitioners as well as to contribute to research dissemination. Also, it provides a helpful guide for those approaching metamorphic testing for the first time. For the validation of the approach, we used the proposed template to describe 17 previously published metamorphic relations from different domains and groups of authors, without finding expressiveness problems. We hope that this work eases the diffusion and adoption of metamorphic testing, contributing to the progress of this thriving testing technique.

[1]  Victor R. Basili,et al.  The TAME Project: Towards Improvement-Oriented Software Environments , 1988, IEEE Trans. Software Eng..

[2]  Rafael Corchuelo,et al.  Supporting requirements verification using XSLT , 2002, Proceedings IEEE Joint International Conference on Requirements Engineering.

[3]  Mikael Lindvall,et al.  Agile Metamorphic Model-Based Testing , 2016, 2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET).

[4]  Antonio Ruiz Cortés,et al.  A Requirements Elicitation Approach Based in Templates and Patterns , 1999, WER.

[5]  Egon Berghout,et al.  The Goal/Question/Metric method: a practical guide for quality improvement of software development , 1999 .

[6]  Baowen Xu,et al.  Testing and validating machine learning classifiers by metamorphic testing , 2011, J. Syst. Softw..

[7]  T. H. Tse,et al.  Finding failures from passed test cases: improving the pattern classification approach to the testing of mesh simplification programs , 2010 .

[8]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[9]  T. H. Tse,et al.  Oracles Are Hardly Attain'd, and Hardly Understood: Confessions of Software Testing Researchers , 2013, 2013 13th International Conference on Quality Software.

[10]  V. Basili Software modeling and measurement: the Goal/Question/Metric paradigm , 1992 .

[11]  Song Huang,et al.  A Formal Model for Metamorphic Relation Decomposition , 2013, 2013 Fourth World Congress on Software Engineering.

[12]  Mikael Lindvall,et al.  Metamorphic Model-Based Testing Applied on NASA DAT -- An Experience Report , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[13]  Sergio Segura,et al.  Automated metamorphic testing of variability analysis tools , 2015, Softw. Test. Verification Reliab..

[14]  Tsong Yueh Chen,et al.  Metamorphic Testing for Software Quality Assessment: A Study of Search Engines , 2016, IEEE Transactions on Software Engineering.

[15]  Tsong Yueh Chen,et al.  Metamorphic Testing: A New Approach for Generating Next Test Cases , 2020, ArXiv.

[16]  Stephen S. Yau,et al.  Testing context-sensitive middleware-based software applications , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[17]  Tsong Yueh Chen,et al.  Metamorphic testing of programs on partial differential equations: a case study , 2002, Proceedings 26th Annual International Computer Software and Applications.

[18]  F.-C. Kuo,et al.  Detecting faults in technical indicator computations for financial market analysis , 2010, The 2nd International Conference on Information Science and Engineering.

[19]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[20]  Tsong Yueh Chen,et al.  Fault-based testing without the need of oracles , 2003, Inf. Softw. Technol..

[21]  Tsong Yueh Chen,et al.  Testing embedded software by metamorphic testing: A wireless metering system case study , 2011, 2011 IEEE 36th Conference on Local Computer Networks.

[22]  Zhendong Su,et al.  Compiler validation via equivalence modulo inputs , 2014, PLDI.

[23]  Jeffrey M. Voas,et al.  Metamorphic Testing for Cybersecurity , 2016, Computer.

[24]  Sergio Segura,et al.  Metamorphic Relation Template v1.0 , 2017 .

[25]  Gail E. Kaiser,et al.  Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles , 2009, 2009 International Conference on Software Testing Verification and Validation.

[26]  Tsong Yueh Chen,et al.  METRIC: METamorphic Relation Identification based on the Category-choice framework , 2016, J. Syst. Softw..