Parallelization of MIN3P-THCm: A high performance computational framework for subsurface flow and reactive transport simulation

Abstract This paper presents the development of ParMIN3P-THCm, a parallel version of the reactive transport code MIN3P-THCm, which can run efficiently on machines ranging from desktop PCs to supercomputers. Parallelization of ParMIN3P-THCm was achieved through the domain decomposition method based on the PETSc library. The code has been developed from the ground up for parallel scalability and has been tested for up to 768 processors with problem sizes up to 100 million unknowns, showing strong scalability in modeling large-scale reactive transport problems. The total speedup tends to be ideal and near linear up to 768 processors when the degrees of freedom per processor is larger than 8000–15,000, depending on the relative complexity of the reactive transport and flow problems. The improved code efficiency allows refining of the model discretization in both space and time and will facilitate 3D simulations that were impractical to carry out with the sequential version of MIN3P-THCm.

[1]  Ming-Hsu Li,et al.  HYDROGEOCHEM: A Coupled Model of Variably Saturated Flow, Thermal Transport, and Reactive Biogeochemical Transport , 2012 .

[2]  K. Ulrich Mayer,et al.  Investigating the role of gas bubble formation and entrapment in contaminated aquifers: Reactive transport modelling. , 2006, Journal of contaminant hydrology.

[3]  Karline Soetaert,et al.  Reactive transport in aquatic ecosystems: Rapid model prototyping in the open source software R , 2012, Environ. Model. Softw..

[4]  G E Hammond,et al.  Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN , 2014, Water resources research.

[5]  Kerry T.B. MacQuarrie,et al.  Solution of the MoMaS reactive transport benchmark with MIN3P—model formulation and simulation results , 2010 .

[6]  Diederik Jacques,et al.  Benchmarks for multicomponent reactive transport across a cement/clay interface , 2015, Computational Geosciences.

[7]  Patrick Amestoy,et al.  A Fully Asynchronous Multifrontal Solver Using Distributed Dynamic Scheduling , 2001, SIAM J. Matrix Anal. Appl..

[8]  Georg Kosakowski,et al.  Decalcification of cracked cement structures , 2015, Computational Geosciences.

[9]  M. V. Genuchten,et al.  Using Texture and Other Soil Properties to Predict the Unsaturated Soil Hydraulic Functions , 1988 .

[10]  K. Mayer,et al.  A numerical model for multicomponent reactive transport in variably saturated porous media , 1999 .

[11]  Richard T Amos,et al.  Investigating ebullition in a sand column using dissolved gas analysis and reactive transport modeling. , 2006, Environmental science & technology.

[12]  Praveen Kumar,et al.  GPU-based high-performance computing for integrated surface-sub-surface flow modeling , 2015, Environ. Model. Softw..

[13]  Pablo Gamazo,et al.  Reactive Transport Modeling of Natural Carbon Sequestration in Ultramafic Mine Tailings , 2012 .

[14]  Javier Samper,et al.  CORE2D V4: A Code for Water Flow, Heat and Solute Transport, Geochemical Reactions, and Microbial Processes , 2012 .

[15]  Kerry T.B. MacQuarrie,et al.  Reactive transport and thermo‐hydro‐mechanical coupling in deep sedimentary basins affected by glaciation cycles: model development, verification, and illustrative example , 2016 .

[16]  Olaf Kolditz,et al.  A parallelization scheme to simulate reactive transport in the subsurface environment with OGS#IPhreeqc 5.5.7-3.1.2 , 2015 .

[17]  James Demmel,et al.  SuperLU_DIST: A scalable distributed-memory sparse direct solver for unsymmetric linear systems , 2003, TOMS.

[18]  B. V. Leer,et al.  Towards the ultimate conservative difference scheme. II. Monotonicity and conservation combined in a second-order scheme , 1974 .

[19]  Mark F. Adams,et al.  Chombo Software Package for AMR Applications Design Document , 2014 .

[20]  Mary F. Wheeler,et al.  Modeling of Flow and Reactive Transport in IPARS , 2012 .

[21]  C. Appelo,et al.  PHT3D: A Reactive Multicomponent Transport Model for Saturated Porous Media , 2010, Ground water.

[22]  R. Maxwell,et al.  Integrated surface-groundwater flow modeling: A free-surface overland flow boundary condition in a parallel groundwater flow model , 2006 .

[23]  Kenneth S. Pitzer,et al.  Thermodynamics of electrolytes. I. Theoretical basis and general equations , 1973 .

[24]  D. A. Barry,et al.  2 , , 1993 .

[25]  Sergi Molins,et al.  Coupling between geochemical reactions and multicomponent gas and solute transport in unsaturated media: A reactive transport modeling study , 2007 .

[26]  D. A. Barry,et al.  A one-dimensional reactive multi-component transport model for biodegradation of petroleum hydrocarbons in groundwater , 1998, Environ. Model. Softw..

[27]  Martinus Oostrom,et al.  STOMP Subsurface Transport Over Multiple Phases, Version 4.0, User’s Guide , 2006 .

[28]  David L. Parkhurst,et al.  Description of input and examples for PHREEQC version 3: a computer program for speciation, batch-reaction, one-dimensional transport, and inverse geochemical calculations , 2013 .

[29]  T. P. Clement and C. D. Johnson.  RT3D: Reactive Transport in 3-Dimensions , 2012 .

[30]  Jarek Nieplocha,et al.  Advances, Applications and Performance of the Global Arrays Shared Memory Programming Toolkit , 2006, Int. J. High Perform. Comput. Appl..

[31]  K. Pruess,et al.  TOUGHREACT: A Simulation Program for Subsurface Reactive Chemical Transport under Non-isothermal Multiphase Flow Conditions , 2012 .

[32]  Vincent Lagneau,et al.  HYTEC results of the MoMas reactive transport benchmark , 2010 .

[33]  Peter A. Forsyth,et al.  A parallel computational framework to solve flow and transport in integrated surface-subsurface hydrologic systems , 2012, Environ. Model. Softw..

[34]  Carl I. Steefel,et al.  ParCrunchFlow: an efficient, parallel reactive transport simulation tool for physically and chemically heterogeneous saturated subsurface environments , 2015, Computational Geosciences.

[35]  J. Meeussen,et al.  ORCHESTRA: an object-oriented framework for implementing chemical equilibrium models. , 2003, Environmental science & technology.

[36]  George Shu Heng Pau,et al.  A high-performance workflow system for subsurface simulation , 2014, Environ. Model. Softw..

[37]  Xiaohui Wei,et al.  THC-MP: High performance numerical simulation of reactive transport and multiphase flow in porous media , 2015, Comput. Geosci..

[38]  J. Islam,et al.  A one-dimensional reactive multi-component landfill leachate transport model , 2002, Environ. Model. Softw..

[39]  Henning Prommer,et al.  A reactive transport benchmark on modeling biogenic uraninite re-oxidation by Fe(III)-(hydr)oxides , 2015, Computational Geosciences.

[40]  William Gropp,et al.  Efficient Management of Parallelism in Object-Oriented Numerical Software Libraries , 1997, SciTools.

[41]  Scott R. Charlton,et al.  Modules based on the geochemical model PHREEQC for use in scripting and programming languages , 2011, Comput. Geosci..

[42]  Joachim Hoffmann,et al.  Comparison of numerical methods for simulating strongly nonlinear and heterogeneous reactive transport problems—the MoMaS benchmark case , 2010 .

[43]  Patrick Amestoy,et al.  Hybrid scheduling for the parallel solution of linear systems , 2006, Parallel Comput..

[44]  Wenqing Wang,et al.  OpenGeoSys: an open-source initiative for numerical simulation of thermo-hydro-mechanical/chemical (THM/C) processes in porous media , 2012, Environmental Earth Sciences.

[45]  Patrick Goblet,et al.  Module-oriented modeling of reactive transport with HYTEC , 2003 .

[46]  Chuan Lu,et al.  PFLOTRAN: Reactive Flow & Transport Code for Use on Laptops to Leadership-Class Supercomputers , 2012 .

[47]  K. Mayer,et al.  Three-dimensional density-dependent flow and multicomponent reactive transport modeling of chlorinated solvent oxidation by potassium permanganate. , 2009, Journal of contaminant hydrology.

[48]  Andrea Toselli,et al.  Domain decomposition methods : algorithms and theory , 2005 .

[49]  David W. Blowes,et al.  Multicomponent reactive transport modeling in variably saturated porous media using a generalized formulation for kinetically controlled reactions , 2002 .

[50]  Patrice de Caritat,et al.  HEAT: a FORTRAN computer program for calculating 1-D conductive and advective heat transport in geological formations , 1993 .

[51]  D. A. Barry,et al.  Implementation of variably saturated flow into PHREEQC for the simulation of biogeochemical reactions in the vadose zone , 2010, Environ. Model. Softw..

[52]  Y. Hao,et al.  Overview of NUFT: A Versatile Numerical Model for Simulating Flow and Reactive Transport in Porous Media , 2012 .

[53]  C. Monnin,et al.  Density calculation and concentration scale conversions for natural waters , 1994 .