System testing in scientific numerical software frameworks using the example of DUNE

We present dune-testtools, a collection of tools for system testing in scientific soft-ware using the example of the Distributed Unified Numerics Environment (Dune). Testing isacknowledged as indispensible support for scientific software development and assurance ofsoftware quality to produce trustworthy simulation results. Most of the time, testing in soft-ware frameworks developed at research facilities is restricted to either unit testing or simplebenchmark programs. However, in a modern numerical software framework, the number of pos-sible feature combinations constituting a program is vast. Only system testing, meaning testingwithin a possible end user environment also emulating variability, can assess software qualityand reproducibility of numerical results. We provide an easy-to-use interface taking workload offdevelopers and administrators in open-source scientific numerical software framework projects.In our approach, the large number of possible combinations is reduced using the scientific expertknowledge of developers to identify the practically relevant combinations. Our approach tosystem testing is designed to be integrated in the workflow of a research software developingscientist.

[1]  David Wells,et al.  The deal.II Library, Version 8.4 , 2016, J. Num. Math..

[2]  Kerstin Mueller,et al.  Software Product Line Engineering Foundations Principles And Techniques , 2016 .

[3]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[4]  Hanna Remmel.  Supporting the quality assurance of a scientific framework , 2014 .

[5]  Andreas Dedner,et al.  A generic grid interface for parallel and adaptive scientific computing. Part I: abstract framework , 2008, Computing.

[6]  Diane Kelly,et al.  Testing for trustworthiness in scientific software , 2009, 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering.

[7]  Barbara Paech,et al.  Supporting the testing of scientific frameworks with software product line engineering: a proposed approach , 2011, SECSE '11.

[8]  T. Trucano,et al.  Verification, Validation, and Predictive Capability in Computational Engineering and Physics , 2004 .

[9]  Steffen Müthing,et al.  DuMux: DUNE for multi-{phase,component,scale,physics,…} flow and transport in porous media , 2011 .

[10]  W. Bangerth,et al.  deal.II—A general-purpose object-oriented finite element library , 2007, TOMS.

[11]  Peter Bastian,et al.  Generic implementation of finite element methods in the Distributed and Unified Numerics Environment (DUNE) , 2010, Kybernetika.

[12]  Ilene Burnstein,et al.  Practical Software Testing: A Process-Oriented Approach , 2003 .

[13]  Anders Logg,et al.  Automated Solution of Differential Equations by the Finite Element Method: The FEniCS Book , 2012 .