Hybrid parallel iterative sparse linear solver framework for reservoir geomechanical and flow simulation

Abstract We discuss new developments of a hybrid parallel iterative sparse linear solver framework focused on petroleum reservoir flow and geomechanical simulation. It runs efficiently on several platforms, from desktop workstations to clusters of multicore nodes, with or without multiple GPUs, using a two-tier hierarchical architecture for distributed matrices and vectors. Results show good parallel scalability. Comparisons with a well-established library and a proprietary commercial solver indicate that our solver is competitive with the best available tools. We present results of the solver's application to simulations of real and synthetic reservoir models of up to billions of unknowns, running on CPUs and GPUs on up to 2,000 processes.

[1]  Hamdi A. Tchelepi,et al.  Parallel Multiscale Linear Solver for Highly Detailed Reservoir Models , 2015, ANSS 2015.

[2]  Michele Benzi,et al.  A Sparse Approximate Inverse Preconditioner for the Conjugate Gradient Method , 1996, SIAM J. Sci. Comput..

[3]  Barry F. Smith,et al.  Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations , 1996 .

[4]  V. E. Henson,et al.  BoomerAMG: a parallel algebraic multigrid solver and preconditioner , 2002 .

[5]  R. Kouhia,et al.  Stabilized and block approximate inverse preconditioners for problems in solid and structural mechanics , 2001 .

[6]  Luiz Mariano Carvalho,et al.  Supporting theory for a block approximate inverse preconditioner , 2021 .

[7]  Roald Brouwer,et al.  Closed Loop Reservoir Management , 2009 .

[8]  Gerard L. G. Sleijpen,et al.  BiCGstab(l) and other hybrid Bi-CG methods , 1994, Numerical Algorithms.

[9]  D. V. Griffiths,et al.  Programming the finite element method , 1982 .

[10]  Ronaldo I. Borja,et al.  Block-preconditioned Newton–Krylov solvers for fully coupled flow and geomechanics , 2011 .

[11]  Massimo Bernaschi,et al.  A Dynamic Pattern Factored Sparse Approximate Inverse Preconditioner on Graphics Processing Units , 2019, SIAM J. Sci. Comput..

[12]  Hans-Joachim Plum,et al.  Status of System-AMG for Reservoir Simulation Applications , 2015, ANSS 2015.

[13]  Qin Lu,et al.  A New Approach To Improve Linear Solver Performance for a Fully Implicit Coupled System of Reservoir and Surface Network , 2013, ANSS 2013.

[14]  Jonathan M. Cohen,et al.  Parallel Graph Coloring with Applications to the Incomplete-LU Factorization on the GPU , 2015 .

[15]  Dirk Schmidl,et al.  Data and thread affinity in openmp programs , 2008, MAW '08.

[16]  Martin J. Blunt,et al.  Tenth SPE Comparative Solution Project: a comparison of upscaling techniques , 2001 .

[17]  John R. Gilbert,et al.  On the representation and multiplication of hypersparse matrices , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing.

[18]  Zhangxin Chen,et al.  Efficient Preconditioning for Algebraic Multigrid and Red-Black Ordering in Adaptive-Implicit Black-Oil Simulations , 2015, ANSS 2015.

[19]  Hamdi A. Tchelepi,et al.  Parallel Scalable Unstructured CPR-Type Linear Solver for Reservoir Simulation , 2005 .

[20]  Hamdi A. Tchelepi,et al.  Scalable Multistage Linear Solver for Coupled Systems of Multisegment Wells and Unstructured Reservoir Models , 2009 .

[21]  John Freeman,et al.  C++ lambda expressions and closures , 2010, Sci. Comput. Program..

[22]  Yuanle Ma,et al.  Computational methods for multiphase flows in porous media , 2007, Math. Comput..

[23]  Olivier Dubrule,et al.  High-Performance Computing: Towards a Safer, Faster and Sharper Exploration & Production , 2014 .

[24]  Victor Eijkhout,et al.  Performance Optimization and Modeling of Blocked Sparse Kernels , 2007, Int. J. High Perform. Comput. Appl..

[25]  Peter A. Forsyth,et al.  Practical considerations for adaptive implicit methods in reservoir simulation , 1986 .

[26]  Sebastian Gries.  On the Convergence of System-AMG in Reservoir Simulation , 2018 .

[27]  Ludmil T. Zikatanov,et al.  Algebraic multigrid methods * , 2016, Acta Numerica.

[28]  Hamdi A. Tchelepi,et al.  Multiscale finite-element method for linear elastic geomechanics , 2017, J. Comput. Phys..

[29]  Jiaquan Gao,et al.  An efficient sparse approximate inverse preconditioning algorithm on GPU , 2020, Concurr. Comput. Pract. Exp..

[30]  D. H. Thurnau,et al.  Reservoir Simulation Using an Adaptive Implicit Method , 1983 .

[31]  Larry S.K. Fung,et al.  From Mega Cell to Giga Cell Reservoir Simulation , 2008 .

[32]  Pradeep Dubey,et al.  Sparsifying Synchronization for High-Performance Shared-Memory Sparse Triangular Solver , 2014, ISC.

[33]  H. Tchelepi,et al.  Multi-scale finite-volume method for elliptic problems in subsurface flow simulation , 2003 .

[34]  Robert Strzodka,et al.  AmgX: A Library for GPU Accelerated Algebraic Multigrid and Preconditioned Iterative Methods , 2015, SIAM J. Sci. Comput..

[35]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[36]  O. C. Zienkiewicz,et al.  VISCO-PLASTICITY--PLASTICITY AND CREEP IN ELASTIC SOLIDS--A UNIFIED NUMERICAL SOLUTION APPROACH , 1974 .

[37]  J. Meijerink,et al.  An iterative solution method for linear systems of which the coefficient matrix is a symmetric -matrix , 1977 .

[38]  K. Aziz,et al.  Petroleum Reservoir Simulation , 1979 .

[39]  Dieter Stolle,et al.  An interpretation of initial stress and strain methods, and numerical stability , 1991 .

[40]  H. Nilsen,et al.  Comparison Between Algebraic Multigrid and Multilevel Multiscale Methods for Reservoir Simulation , 2020 .

[41]  Barbara Chapman,et al.  Using OpenMP - portable shared memory parallel programming , 2007, Scientific and engineering computation.

[42]  Hamdi A. Tchelepi,et al.  Multiscale two-stage solver for Biot’s poroelasticity equations in subsurface media , 2018, Computational Geosciences.

[43]  Henk A. van der Vorst,et al.  Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems , 1992, SIAM J. Sci. Comput..

[44]  C. K. Filelis-Papadopoulos,et al.  Hybrid multi-projection method using sparse approximate inverses on GPU clusters , 2020, Int. J. High Perform. Comput. Appl..

[45]  Larry S.K. Fung,et al.  A Parallel Reservoir Simulator for Large-Scale Reservoir Simulation , 2002 .

[46]  R. P. Kendall,et al.  Constrained Residual Acceleration of Conjugate Residual Methods , 1985 .

[47]  John Killough,et al.  Parallel Linear Solvers for Reservoir Simulation: A Generic Approach for Existing and Emerging Computer Architectures , 1994 .

[48]  Sunita Chandrasekaran,et al.  OpenACC for Programmers: Concepts and Strategies , 2017 .

[49]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[50]  David A. Collins,et al.  A Shared-Memory Parallel Black-Oil Simulator with a Parallel ILU Linear Solver , 2003 .

[51]  Maxim Naumov,et al.  Parallel Solution of Sparse Triangular Linear Systems in the Preconditioned Iterative Methods on the GPU , 2011 .

[52]  Bret L. Beckner,et al.  An Efficient Multicore Linear Solver for Reservoir Simulation Applications , 2013, ANSS 2013.

[53]  Denis Demidov,et al.  AMGCL: An Efficient, Flexible, and Extensible Algebraic Multigrid Implementation , 2018, Lobachevskii Journal of Mathematics.

[54]  Yousef Saad,et al.  A Flexible Inner-Outer Preconditioned GMRES Algorithm , 1993, SIAM J. Sci. Comput..

[55]  Bjarne Foss,et al.  Closed-Loop Predictions in Reservoir Management Under Uncertainty , 2017 .

[56]  Luiz Mariano Carvalho,et al.  A Linear Solver Framework for Flow and Geomechanics Reservoir Simulation , 2019, 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW).

[57]  Yousef Saad,et al.  GPU-accelerated preconditioned iterative linear solvers , 2013, The Journal of Supercomputing.

[58]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[59]  I. Cormeau,et al.  Numerical stability in quasi‐static elasto/visco‐plasticity , 1975 .

[60]  Kamy Sepehrnoori,et al.  Methodology for geomechanically controlled transmissibility through active natural fractures in reservoir simulation , 2016 .

[61]  Duc H. Le,et al.  An Adaptive Ensemble Smoother with Multiple Data Assimilation for Assisted History Matching , 2015, ANSS 2015.