A Distributed-Memory Parallelization of a Shared-Memory Parallel Ensemble Kalman Filter

Inverse problems arise in various areas of science and engineering. These problems are not only difficult to solve numerically, but they also require a large amount of computer resources both in time and memory. It is therefore not surprising that inverse problems are often solved using techniques from high-performance computing. We consider the parallelization of an inverse problem in the field of geothermal reservoir engineering. In this particular scientific application, the underlying software package is already parallelized using the shared-memory programming paradigm Open MP. Here, we present an extension of this parallelization to distributed memory enabling a hybrid Open MP/MPI parallelization. The situation is different from the standard way of hybrid parallel programming because the data structures of the Open MP-parallelized code differ from those in the serial implementation. We exploit this transformation of the data structures in our distributed-memory strategy for parallelizing an ensemble Kalman filter, a particular method for the solution of inverse problems. We describe this novel parallelization strategy, introduce a performance model, and present timing results on a compute cluster using nodes with 2 sockets, each equipped with Intel Xeon X5675 Westmere EP processors with 6 cores. All timing results are obtained with a pure MPI parallelization without using any Open MP threads.

[1]  Andreas Wolf,et al.  Ein Softwarekonzept zur hierarchischen Parallelisierung von stochastischen und deterministischen Inversionsproblemen auf modernen ccNUMA-Plattformen unter Nutzung automatischer Programmtransformation , 2011 .

[2]  D. Oliver,et al.  Recent progress on reservoir history matching: a review , 2011 .

[3]  Clayton V. Deutsch,et al.  GSLIB: Geostatistical Software Library and User's Guide , 1993 .

[4]  Alexandre Boucher,et al.  Applied Geostatistics with SGeMS: A User's Guide , 2009 .

[5]  Chia-Jung Hsu.  Numerical Heat Transfer and Fluid Flow , 1981 .

[6]  D. Stensrud,et al.  The Ensemble Kalman Filter for Combined State and Parameter Estimation , 2009 .

[7]  Eugenia Kalnay,et al.  Atmospheric Modeling, Data Assimilation and Predictability , 2002 .

[8]  J. Chilès,et al.  Geostatistics: Modeling Spatial Uncertainty , 1999 .

[9]  Louis B. Rall,et al.  Automatic Differentiation: Techniques and Applications , 1981, Lecture Notes in Computer Science.

[10]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[11]  C. Clauser,et al.  Optimization of geothermal energy reservoir modeling using advanced numerical tools for stochastic parameter estimation and quantifying uncertainties , 2013 .

[12]  M. Kühn,et al.  Numerical Simulation of Reactive Flow using SHEMAT , 2003 .

[13]  C. Vogel.  Computational Methods for Inverse Problems , 1987 .

[14]  Ning Liu,et al.  Inverse Theory for Petroleum Reservoir Characterization and History Matching , 2008 .

[15]  G. Evensen.  Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics , 1994 .

[16]  Jean-Paul Chilbs,et al.  Geostatistics , 2000, Technometrics.

[17]  Christina Freytag,et al.  Using Mpi Portable Parallel Programming With The Message Passing Interface , 2016 .

[18]  Dean S. Oliver,et al.  THE ENSEMBLE KALMAN FILTER IN RESERVOIR ENGINEERING-A REVIEW , 2009 .

[19]  Alexandre Boucher,et al.  Applied Geostatistics with SGeMS: Preface , 2009 .

[20]  G. Evensen.  The ensemble Kalman filter for combined state and parameter estimation , 2009, IEEE Control Systems.

[21]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[22]  Philippe Courtier,et al.  Unified Notation for Data Assimilation : Operational, Sequential and Variational , 1997 .

[23]  R. E. Kalman,et al.  New Results in Linear Filtering and Prediction Theory , 1961 .

[24]  H. Martin Bücker,et al.  Preservation of non-uniform memory architecture characteristics when going from a nested OpenMP to a hybrid MPI/OpenMP approach , 2014, 2014 4th International Conference On Simulation And Modeling Methodologies, Technologies And Applications (SIMULTECH).

[25]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[26]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[27]  G. Evensen.  Data Assimilation: The Ensemble Kalman Filter , 2006 .

[28]  B. Khattatov,et al.  Data assimilation : making sense of observations , 2010 .

[29]  Christian Vogt,et al.  On self-potential data for estimating permeability in enhanced geothermal systems , 2014 .

[30]  Christian Vogt,et al.  Estimating the permeability distribution and its uncertainty at the EGS demonstration reservoir Soultz‐sous‐Forêts using the ensemble Kalman filter , 2012 .

[31]  D. Lynch.  Numerical Partial Differential Equations for Environmental Scientists and Engineers: A First Practical Course , 2004 .

[32]  H. M. Bücker,et al.  Joint three-dimensional inversion of coupled groundwater flow and heat transfer based on automatic differentiation: sensitivity calculation, verification, and synthetic examples , 2006 .

[33]  Clayton V. Deutsch,et al.  Geostatistical Software Library and User's Guide , 1998 .

[34]  Geir Evensen,et al.  The Ensemble Kalman Filter: theoretical formulation and practical implementation , 2003 .

[35]  G. Evensen.  Using the Extended Kalman Filter with a Multilayer Quasi-Geostrophic Ocean Model , 1992 .

[36]  Albert Tarantola,et al.  Inverse problem theory - and methods for model parameter estimation , 2004 .

[37]  Samuli Siltanen,et al.  Linear and Nonlinear Inverse Problems with Practical Applications , 2012, Computational science and engineering.

[38]  John L. Gustafson,et al.  Reevaluating Amdahl's law , 1988, CACM.

[39]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.

[40]  S. Lakshmivarahan,et al.  Ensemble Kalman filter , 2009, IEEE Control Systems.

[41]  Clifford H. Thurber,et al.  Parameter estimation and inverse problems , 2005 .