Toward model-driven engineering principles and practices for model replicability and experiment reproducibility

Recent years have seen a proliferation of the use of simulation models in computational science. Most of these models have never been independently replicated by anyone but the original developer. Furthermore, there is a growing credibility gap due to widespread, relaxed attitudes in communication of experiments, models, and validation of simulations used in computational research. We examine various issues and challenges involved in model replication and simulation experiment reproducibility. Model-driven simulation engineering principles and model transformation concepts are adopted as solution strategies to improve replicability of models and reproducibility of experiments. A process model, an architectural framework, and an implementation strategy is introduced to address identified issues in simulation experiment management and model replication.

[1]  Jennifer M. Urban,et al.  Shining Light into Black Boxes , 2012, Science.

[2]  Carole A. Goble,et al.  The design and realisation of the myExperiment Virtual Research Environment for social sharing of workflows , 2009, Future Gener. Comput. Syst..

[3]  Juliana Freire,et al.  Provenance and scientific workflows: challenges and opportunities , 2008, SIGMOD Conference.

[4]  Jill P Mesirov,et al.  Accessible Reproducible Research , 2010, Science.

[5]  Steffen Mazanek,et al.  SHARE: a web portal for creating and sharing executable research papers , 2011, ICCS.

[6]  Tom Mens,et al.  A Taxonomy of Model Transformation , 2006, GRaMoT@GPCE.

[7]  Colin Atkinson,et al.  Model-Driven Development: A Metamodeling Foundation , 2003, IEEE Softw..

[8]  Jacky L. Snoep,et al.  Reproducible computational biology experiments with SED-ML - The Simulation Experiment Description Markup Language , 2011, BMC Systems Biology.

[9]  Douglas C. Schmidt,et al.  Guest Editor's Introduction: Model-Driven Engineering , 2006, Computer.

[10]  Philippe Bonnet,et al.  A Provenance-Based Infrastructure to Support the Life Cycle of Executable Papers , 2011, ICCS.

[11]  Philippe Bonnet,et al.  Exploring the Coming Repositories of Reproducible Experiments: Challenges and Opportunities , 2011, Proc. VLDB Endow..

[12]  Jean Bézivin,et al.  On the Need for Megamodels , 2004, OOPSLA 2004.

[13]  Roger D Peng,et al.  Reproducible research and Biostatistics. , 2009, Biostatistics.

[14]  Matthew R. Pocock,et al.  Taverna: a tool for the composition and enactment of bioinformatics workflows , 2004, Bioinform..

[15]  David L. Donoho,et al.  A Universal Identifier for Computational Results , 2011, ICCS.

[16]  Arian Maleki,et al.  Reproducible Research in Computational Harmonic Analysis , 2009, Computing in Science & Engineering.

[17]  Sergey Fomel,et al.  Reproducible Computational Experiments using Scons , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[18]  Richmond T. Bell The scientific method in practice , 1931 .

[19]  Piotr Nowakowski,et al.  The Collage Authoring Environment , 2011, ICCS.