Hierarchical Metamorphic Relations for Testing Scientific Software

Scientist developers have not yet routinely adopted systematic testing techniques to assure software quality. A key challenge is the oracle problem, a situation in which appropriate mechanisms are unavailable for checking if the code produces the expected output when executed using a set of test cases (TCs). Metamorphic testing alleviates the oracle problem by specifying the relationship that a source TC and its follow-up TC shall meet. Such relationships are called metamorphic relations (MRs) which are necessary properties of the intended program's functionality. Existing approaches handle the MRs in a flat manner. This paper introduces a novel way to facilitate a hierarchy of MRs to be developed incrementally. We illustrate our approach by testing U.S. EPA's Storm Water Management Model (SWMM). The results offer concrete insights into developing effective MRs to systematically test scientific software.

[1]  Robert M. Hierons.  Oracles for Distributed Testing , 2012, IEEE Transactions on Software Engineering.

[2]  Diane Kelly,et al.  Software Engineering for Scientists , 2011, Comput. Sci. Eng..

[3]  Sjaak Brinkkemper,et al.  Requirements Engineering and Continuous Deployment , 2018, IEEE Software.

[4]  William Pao,et al.  Metamorphic Testing Using Geometric Interrogation Technique and Its Application , 1970 .

[5]  S. L. Eddins,et al.  Automated Software Testing for Matlab , 2009, Computing in Science & Engineering.

[6]  Tsong Yueh Chen,et al.  Testing an Open Source Suite for Open Queuing Network Modelling Using Metamorphic Testing Technique , 2009, 2009 14th IEEE International Conference on Engineering of Complex Computer Systems.

[7]  Tsong Yueh Chen,et al.  Metamorphic Testing: A New Approach for Generating Next Test Cases , 2020, ArXiv.

[8]  Jeffrey C. Carver,et al.  Software Development Environments for Scientific and Engineering Software: A Series of Case Studies , 2007, 29th International Conference on Software Engineering (ICSE'07).

[9]  Jesse H. Poore,et al.  Modeling Input Space for Testing Scientific Computational Software: A Case Study , 2008, ICCS.

[10]  Stephen McCamant,et al.  The Daikon system for dynamic detection of likely invariants , 2007, Sci. Comput. Program..

[11]  Tsong Yueh Chen,et al.  Metamorphic testing of programs on partial differential equations: a case study , 2002, Proceedings 26th Annual International Computer Software and Applications.

[12]  Xiaoyu Jin,et al.  A Novel Coupling Pattern in Computational Science and Engineering Software , 2017, 2017 IEEE/ACM 12th International Workshop on Software Engineering for Science (SE4Science).

[13]  Michael K. Stenstrom,et al.  Automatic Calibration of the U.S. EPA SWMM Model for a Large Urban Catchment , 2008 .

[14]  Diane Kelly,et al.  Five Recommended Practices for Computational Scientists Who Write Software , 2009, Computing in Science & Engineering.

[15]  Konstantin Kreyman,et al.  Inspection Procedures for Critical Programs that Model Physical Phenomena , 2001 .

[16]  Dianxiang Xu,et al.  Metamorphic testing of a Monte Carlo modeling program , 2011, AST '11.

[17]  Xiaoyu Jin,et al.  Facilitating end-user developers by estimating time cost of foraging a webpage , 2017, 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[18]  Linda Newman,et al.  Advancing viewpoint merging in requirements engineering: a theoretical replication and explanatory study , 2017, Requirements Engineering.

[19]  Xiaoyu Jin,et al.  Pragmatic Software Reuse in Bioinformatics: How Can Social Network Information Help? , 2016, ICSR.

[20]  Diane Kelly,et al.  Dealing with Risk in Scientific Software Development , 2008, IEEE Software.

[21]  Xiaoyu Jin,et al.  On the impact of social network information diversity on end-user programming productivity: a foraging-theoretic study , 2016, SSE@SIGSOFT FSE.

[22]  Jeffrey C. Carver,et al.  Understanding the High-Performance-Computing Community: A Software Engineer's Perspective , 2008, IEEE Software.

[23]  James M. Bieman,et al.  Testing scientific software: A systematic literature review , 2014, Inf. Softw. Technol..

[24]  Tsong Yueh Chen,et al.  Metamorphic Testing for Software Quality Assessment: A Study of Search Engines , 2016, IEEE Transactions on Software Engineering.

[25]  Gail E. Kaiser,et al.  Empirical Evaluation of Approaches to Testing Applications without Test Oracles , 2010 .

[26]  Judith Segal,et al.  Some Challenges Facing Scientific Software Developers: The Case of Molecular Biology , 2009, 2009 Fifth IEEE International Conference on e-Science.

[27]  Kourtis I.M.  Calibration and validation of SWMM model in two urban catchments in Athens , Greece , 2017 .

[28]  Zhendong Niu,et al.  A Systems Approach to Product Line Requirements Reuse , 2014, IEEE Systems Journal.

[29]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[30]  Huai Liu,et al.  Conformance Testing of Network Simulators Based on Metamorphic Testing Technique , 2009, FMOODS/FORTE.

[31]  Linda Newman,et al.  Advancing Repeated Research in Requirements Engineering: A Theoretical Replication of Viewpoint Merging , 2016, 2016 IEEE 24th International Requirements Engineering Conference (RE).

[32]  Jeffrey C. Carver,et al.  What Scientists and Engineers Think They Know About Software Engineering : A Survey , 2011 .

[33]  J. Suárez,et al.  Stormwater quality calibration by SWMM : a case study in Northern Spain , 2007 .

[34]  Dongmei Zhang,et al.  An Application of Metamorphic Testing for Testing Scientific Software , 2016, 2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET).

[35]  Douglass E. Post,et al.  Software Project Management and Quality Engineering Practices for Complex, Coupled Multiphysics, Massively Parallel Computational Simulations: Lessons Learned From ASCI , 2004, Int. J. High Perform. Comput. Appl..