VECMAtk: a scalable verification, validation and uncertainty quantification toolkit for scientific simulations

We present the VECMA toolkit (VECMAtk), a flexible software environment for single and multiscale simulations that introduces directly applicable and reusable procedures for verification, validation (V&V), sensitivity analysis (SA) and uncertainty quantification (UQ). It enables users to verify key aspects of their applications, systematically compare and validate the simulation outputs against observational or benchmark data, and run simulations conveniently on any platform from the desktop to current multi-petascale computers. In this sequel to our paper on VECMAtk which we presented last year, we focus on a range of functional and performance improvements that we have introduced, cover newly introduced components, and applications examples from seven different domains such as conflict modelling and environmental sciences. We also present several implemented patterns for UQ/SA and V&V, and guide the reader through one example concerning COVID-19 modelling in detail.

[1]  Alfons G. Hoekstra,et al.  Foundations of distributed multiscale computing: Formalization, specification, and analysis , 2013, J. Parallel Distributed Comput..

[2]  Gaute T. Einevoll,et al.  Uncertainpy: A Python Toolbox for Uncertainty Quantification and Sensitivity Analysis in Computational Neuroscience , 2018, bioRxiv.

[3]  James D. Brown,et al.  The Data Uncertainty Engine (DUE): A software tool for assessing and simulating uncertain environmental variables , 2007, Comput. Geosci..

[4]  Peter V. Coveney,et al.  Multiscale computing in the exascale era , 2016, J. Comput. Sci..

[5]  Bertrand Iooss,et al.  Title: Open TURNS: An industrial software for uncertainty quantification in simulation , 2015, 1501.05242.

[6]  Wouter Edeling,et al.  Reducing data-driven dynamical subgrid scale models by physical constraints , 2020 .

[7]  Hamid Arabnejad,et al.  Introducing VECMAtk - Verification, Validation and Uncertainty Quantification for Multiscale and HPC Simulations , 2019, ICCS.

[8]  Robert B. Gramacy,et al.  Practical Heteroscedastic Gaussian Process Modeling for Large Simulation Experiments , 2016, Journal of Computational and Graphical Statistics.

[9]  James R. Gattiker,et al.  Gaussian Process-Based Sensitivity Analysis and Bayesian Model Calibration with GPMSA , 2017 .

[10]  Samantha S. Foley,et al.  Many-task applications in the Integrated Plasma Simulator , 2010, 2010 3rd Workshop on Many-Task Computing on Grids and Supercomputers.

[11]  Derek Groen,et al.  Simulating Refugee Movements: Where Would You Go? , 2016, ICCS.

[12]  Daniel S. Katz,et al.  Swift/T: Large-Scale Application Composition via Distributed-Memory Dataflow Processing , 2013, 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing.

[13]  Peter V Coveney,et al.  Principles Governing Control of Aggregation and Dispersion of Graphene and Graphene Oxide in Polymer Melts , 2020, Advanced materials.

[14]  Leonard E. Schwer,et al.  An overview of the PTC 60/V&V 10: guide for verification and validation in computational solid mechanics , 2007, Engineering with Computers.

[15]  Anirban Mondal,et al.  Stochastic Simulators: An Overview with Opportunities , 2020 .

[16]  David Bell,et al.  Towards an automated framework for agent-based simulation of refugee movements , 2017, 2017 Winter Simulation Conference (WSC).

[17]  Alfons G. Hoekstra,et al.  Semi-intrusive uncertainty propagation for multiscale models , 2018, J. Comput. Sci..

[18]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[19]  Samantha S. Foley,et al.  Parameter Sweep and Optimization of Loosely Coupled Simulations Using the DAKOTA Toolkit , 2012, 2012 IEEE 15th International Conference on Computational Science and Engineering.

[20]  Derek Groen,et al.  A generalized simulation development approach for predicting refugee destinations , 2017, Scientific Reports.

[21]  Anna Nikishova,et al.  Uncertainty Quantification of a Multiscale Model for In-Stent Restenosis , 2018, Cardiovascular engineering and technology.

[22]  Simon J. E. Taylor,et al.  Towards Modelling the Effect of Evolving Violence on Forced Migration , 2019, 2019 Winter Simulation Conference (WSC).

[23]  Christopher J. Roy,et al.  Verification and Validation in Scientific Computing , 2010 .

[24]  Alfons G. Hoekstra,et al.  Easing Multiscale Model Design and Coupling with MUSCLE 3 , 2020, ICCS.

[25]  C. Villani.  Optimal Transport: Old and New , 2008 .

[26]  A G Hoekstra,et al.  Semi-intrusive multiscale metamodelling uncertainty quantification with application to a model of in-stent restenosis , 2019, Philosophical Transactions of the Royal Society A.

[27]  J A Hittinger,et al.  Uncertainty Quantification in the Fusion Simulation Project Verification and Validation Activity , 2010 .

[28]  Stefano Marelli,et al.  UQLab: a framework for uncertainty quantification in MATLAB , 2014 .

[29]  Wei Chen,et al.  Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification , 2012 .

[30]  John Salvatier,et al.  Probabilistic programming in Python using PyMC3 , 2016, PeerJ Comput. Sci..

[31]  Derek Groen,et al.  How Policy Decisions Affect Refugee Journeys in South Sudan: A Study Using Automated Ensemble Simulations , 2020, J. Artif. Soc. Soc. Simul..

[32]  Andrzej A. Wyszogrodzki,et al.  Evaluation of the coupling between mesoscale-WRF and LES‐EULAG models for simulating fine-scale urban dispersion , 2012 .

[33]  Christopher J. Roy,et al.  A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing , 2011 .

[34]  R. A. Richardson,et al.  The heterogeneous multiscale method applied to inelastic polymer mechanics , 2019, Philosophical Transactions of the Royal Society A.

[35]  Francesca Pianosi,et al.  A Matlab toolbox for Global Sensitivity Analysis , 2015, Environ. Model. Softw..

[36]  Michael S. Eldred,et al.  DAKOTA , A Multilevel Parallel Object-Oriented Framework for Design Optimization , Parameter Estimation , Uncertainty Quantification , and Sensitivity Analysis Version 4 . 0 User ’ s Manual , 2006 .

[37]  Wouter Edeling,et al.  Resampling with neural networks for stochastic parameterization in multiscale systems , 2020, Physica D: Nonlinear Phenomena.

[38]  Anna Nikishova,et al.  Non-intrusive and semi-intrusive uncertainty quantification of a multiscale in-stent restenosis model , 2020, Reliab. Eng. Syst. Saf..

[39]  Shantenu Jha,et al.  RADICAL-Cybertools: Middleware Building Blocks for Scalable Science , 2019, ArXiv.

[40]  David W. Engel,et al.  Survey and Evaluate Uncertainty Quantification Methodologies , 2012 .

[41]  Ian Foster,et al.  Parsl: Pervasive Parallel Programming in Python , 2019, HPDC.

[42]  Jalal Lakhlili,et al.  EasyVVUQ: A Library for Verification, Validation and Uncertainty Quantification in High Performance Computing , 2020 .

[43]  Huaiyu Zhu.  On Information and Sufficiency , 1997 .

[44]  Joris Borgdorff,et al.  A framework for multi-scale modelling , 2014, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[45]  A. P. Siebesma,et al.  Formulation of the Dutch Atmospheric Large-Eddy Simulation (DALES) and overview of its applications , 2010 .

[46]  Michael S. Eldred,et al.  DAKOTA : a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty quantification, and sensitivity analysis. Version 5.0, user's reference manual. , 2010 .

[47]  J. Prusa,et al.  EULAG, a computational model for multiscale flows , 2008 .

[48]  Jan Weglarz,et al.  Building Confidence in Simulation: Applications of EasyVVUQ , 2020, Advanced Theory and Simulations.

[49]  Stefan Heinrich,et al.  Multilevel Monte Carlo Methods , 2001, LSSC.

[50]  Jo Graham,et al.  Old and new , 2000 .

[51]  Peter V. Coveney,et al.  Toward High Fidelity Materials Property Prediction from Multiscale Modeling and Simulation , 2019, Advanced Theory and Simulations.

[52]  The Primer , 1906 .

[53]  Shantenu Jha,et al.  Towards a common model for pilot-jobs , 2012, HPDC '12.

[54]  Jalal Lakhlili,et al.  Uncertainty Quantification for Multiscale Fusion Plasma Simulations with VECMA Toolkit , 2020, ICCS.

[55]  Olivier Hoenen,et al.  ComPat framework for multiscale simulations applied to fusion plasmas , 2019, Comput. Phys. Commun..

[56]  P. Gent,et al.  Isopycnal mixing in ocean circulation models , 1990 .

[57]  Peter V. Coveney,et al.  FabSim: Facilitating computational research through automation on large-scale and distributed e-infrastructures , 2015, Comput. Phys. Commun..

[58]  Todd Simmermacher,et al.  The Role of Model V&V in the Defining of Specifications , 2015 .

[59]  Richard Mateosian,et al.  Old and New , 2006, IEEE Micro.