Unit Tests of Scientific Software: A Study on SWMM

Testing helps assure software quality by executing program and uncovering bugs. Scientific software developers often find it challenging to carry out systematic and automated testing due to reasons like inherent model uncertainties and complex floating point computations. We report in this paper a manual analysis of the unit tests written by the developers of the Storm Water Management Model (SWMM). The results show that the 1,458 SWMM tests have a 54.0% code coverage and a 82.4% user manual coverage. We also observe a “getter-setter-getter” testing pattern from the SWMM unit tests. Based on these results, we offer insights to improve test development and coverage.

[1]  Dongmei Zhang,et al.  An Application of Metamorphic Testing for Testing Scientific Software , 2016, 2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET).

[2]  Nan Niu,et al.  Tagging in Assisted Tracing , 2015, 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability.

[3]  Nan Niu,et al.  TraCter: A tool for candidate traceability link clustering , 2011, 2011 IEEE 19th International Requirements Engineering Conference.

[4]  Johannes Mayer,et al.  On Testing Image Processing Applications with Statistical Methods , 2005, Software Engineering.

[5]  Linda Newman,et al.  Advancing Repeated Research in Requirements Engineering: A Theoretical Replication of Viewpoint Merging , 2016, 2016 IEEE 24th International Requirements Engineering Conference (RE).

[6]  Xiaoyu Jin,et al.  A Novel Coupling Pattern in Computational Science and Engineering Software , 2017, 2017 IEEE/ACM 12th International Workshop on Software Engineering for Science (SE4Science).

[7]  Nan Niu,et al.  Supporting requirements to code traceability through refactoring , 2013, Requirements Engineering.

[8]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[9]  Nachiappan Nagappan,et al.  Evaluating the efficacy of test-driven development: industrial case studies , 2006, ISESE '06.

[10]  James M. Bieman,et al.  Testing scientific software: A systematic literature review , 2014, Inf. Softw. Technol..

[11]  Nan Niu,et al.  Deep Learning for Smart Sewer Systems: Assessing Nonfunctional Requirements , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS).

[12]  Nan Niu,et al.  Hierarchical Metamorphic Relations for Testing Scientific Software , 2018, 2018 IEEE/ACM 13th International Workshop on Software Engineering for Science (SE4Science).

[13]  M. Mrowiec,et al.  Modelling of green roofs' hydrologic performance using EPA's SWMM. , 2013, Water science and technology : a journal of the International Association on Water Pollution Research.

[14]  Nan Niu,et al.  ReCVisu: A tool for clustering-based visual exploration of requirements , 2012, 2012 20th IEEE International Requirements Engineering Conference (RE).

[15]  Sandeep Koranne,et al.  Boost C++ Libraries , 2011 .

[16]  Sjaak Brinkkemper,et al.  Requirements Engineering and Continuous Deployment , 2018, IEEE Software.

[17]  Zhendong Niu,et al.  Automatically Tracing Dependability Requirements via Term-Based Relevance Feedback , 2018, IEEE Transactions on Industrial Informatics.

[18]  Michelle Simon,et al.  Releasing Scientific Software in GitHub: A Case Study on SWMM2PEST , 2019, 2019 IEEE/ACM 14th International Workshop on Software Engineering for Science (SE4Science).

[19]  David A. Ham,et al.  Automated continuous verification for numerical simulation , 2011 .

[20]  Nan Niu,et al.  Gray links in the use of requirements traceability , 2016, SIGSOFT FSE.

[21]  Linda Newman,et al.  Advancing viewpoint merging in requirements engineering: a theoretical replication and explanatory study , 2017, Requirements Engineering.

[22]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[23]  Nancy G. Leveson,et al.  Analysis of Faults in an N-Version Software Experiment , 1990, IEEE Trans. Software Eng..

[24]  Paul F. Dubois.  Testing Scientific Programs , 2012, Computing in Science & Engineering.

[25]  Kourtis I.M.  Calibration and validation of SWMM model in two urban catchments in Athens , Greece , 2017 .

[26]  Steve M. Easterbrook,et al.  Engineering the Software for Understanding Climate Change , 2009, Computing in Science & Engineering.

[27]  Zhendong Niu,et al.  Complementarity in Requirements Tracing , 2020, IEEE Transactions on Cybernetics.

[28]  Marco Kuhrmann,et al.  Is 100% Test Coverage a Reasonable Requirement? Lessons Learned from a Space Software Project , 2017, PROFES.

[29]  Jeffrey C. Carver,et al.  Exploratory Metamorphic Testing for Scientific Software , 2018, Computing in Science & Engineering.