Integration of adversarial autoencoders with residual dense convolutional networks for inversion of solute transport in non-Gaussian conductivity fields

Characterization of a non-Gaussian channelized conductivity field in subsurface flow and transport modeling through inverse modeling usually leads to a high-dimensional inverse problem and requires repeated evaluations of the forward model. In this study, we develop a convolutional adversarial autoencoder (CAAE) network to parameterize the high-dimensional non-Gaussian conductivity fields using a low-dimensional latent representation and a deep residual dense convolutional network (DRDCN) to efficiently construct a surrogate model for the forward model. The two networks are both based on a multilevel residual learning architecture called residual-in-residual dense block. The multilevel residual learning strategy and the dense connection structure in the dense block ease the training of deep networks, enabling us to efficiently build deeper networks that have an essentially increased capacity for approximating mappings of very high-complexity. The CCAE and DRDCN networks are incorporated into an iterative local updating ensemble smoother to formulate an inversion framework. The integrated method is demonstrated using a synthetic solute transport model. Results indicate that CAAE is a robust parameterization method for the channelized conductivity fields with Gaussian conductivities within each facies. The DRDCN network is able to obtain an accurate surrogate model of the forward model with high-dimensional and highly-complex concentration fields using relatively limited training data. The CAAE paramterization approach and the DRDCN surrogate method together significantly reduce the number of forward model runs required to achieve accurate inversion results.

[1]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Louis J. Durlofsky,et al.  A New Differentiable Parameterization Based on Principal Component Analysis for the Low-Dimensional Representation of Complex Geological Models , 2014, Mathematical Geosciences.

[3]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[4]  Teng Xu,et al.  Characterization of non‐Gaussian conductivities and porosities with hydraulic heads, solute concentrations, and water temperatures , 2016 .

[5]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[6]  Cheng Chen,et al.  Efficient Bayesian experimental design for contaminant source identification , 2015, Water Resources Research.

[7]  Diganta Misra,et al.  Mish: A Self Regularized Non-Monotonic Neural Activation Function , 2019, ArXiv.

[8]  Jichun Wu,et al.  Deep Autoregressive Neural Networks for High‐Dimensional Inverse Problems in Groundwater Contaminant Source Identification , 2018, Water Resources Research.

[9]  Behnam Jafarpour Wavelet Reconstruction of Geologic Facies From Nonlinear Dynamic Flow Measurements , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[10]  A. Journel,et al.  Entropy and spatial disorder , 1993 .

[11]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  G. Evensen,et al.  Data assimilation and inverse methods in terms of a probabilistic formulation , 1996 .

[13]  Eric Laloy,et al.  Inversion using a new low-dimensional representation of complex binary geological media based on a deep neural network , 2017, 1710.09196.

[14]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Jian Wang,et al.  Deep learning and its application in geochemical mapping , 2019, Earth-Science Reviews.

[16]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[17]  Arlen W. Harbaugh,et al.  MODFLOW-2000, The U.S. Geological Survey Modular Ground-Water Model - User Guide to Modularization Concepts and the Ground-Water Flow Process , 2000 .

[18]  J. Gómez-Hernández,et al.  To be or not to be multi-Gaussian? A reflection on stochastic hydrogeology , 1998 .

[19]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[21]  T. Hansen,et al.  Inverse problems with non-trivial priors: efficient solution through sequential Gibbs sampling , 2012, Computational Geosciences.

[22]  D. Oliver,et al.  Recent progress on reservoir history matching: a review , 2011 .

[23]  D. Mallants,et al.  Efficient posterior exploration of a high‐dimensional groundwater model from two‐stage Markov chain Monte Carlo simulation and polynomial chaos expansion , 2013 .

[24]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[25]  Paris Perdikaris,et al.  Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data , 2019, J. Comput. Phys..

[26]  Marco Aurélio Cavalcanti Pacheco,et al.  History matching geological facies models based on ensemble smoother and deep generative models , 2019, Journal of Petroleum Science and Engineering.

[27]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[28]  Gregoire Mariethoz,et al.  Merging parallel tempering with sequential geostatistical resampling for improved posterior exploration of high-dimensional subsurface categorical fields , 2016 .

[29]  Ahmed H. Elsheikh,et al.  Reduced-Order Modeling of Subsurface Multi-phase Flow Models Using Deep Residual Recurrent Neural Networks , 2018, Transport in Porous Media.

[30]  Guannan Zhang,et al.  A Taylor Expansion‐Based Adaptive Design Strategy for Global Surrogate Modeling With Applications in Groundwater Modeling , 2017 .

[31]  Dongxiao Zhang,et al.  An efficient, high-order perturbation approach for flow in random porous media via Karhunen-Loève and polynomial expansions , 2004 .

[32]  Tiangang Cui,et al.  Bayesian calibration of a large‐scale geothermal reservoir model by a new adaptive delayed acceptance Metropolis Hastings algorithm , 2011 .

[33]  Liangping Li,et al.  The power of transient piezometric head data in inverse modeling: An application of the localized normal-score EnKF with covariance inflation in a heterogenous bimodal hydraulic conductivity field , 2013 .

[34]  Marco Aurélio Cavalcanti Pacheco,et al.  Towards a Robust Parameterization for Conditioning Facies Models Using Deep Variational Autoencoders and Ensemble Smoother , 2018, Comput. Geosci..

[35]  N. Linde,et al.  Image synthesis with graph cuts: a fast model proposal mechanism in probabilistic inversion , 2016, 1701.01593.

[36]  Jasper A. Vrugt,et al.  Markov chain Monte Carlo simulation using the DREAM software package: Theory, concepts, and MATLAB implementation , 2016, Environ. Model. Softw..

[37]  Daniel M. Tartakovsky,et al.  Moment Differential Equations for Flow in Highly Heterogeneous Porous Media , 2003 .

[38]  Hamdi A. Tchelepi,et al.  A two-stage adaptive stochastic collocation method on nested sparse grids for multiphase flow in randomly heterogeneous porous media , 2017, J. Comput. Phys..

[39]  Bryan A. Tolson,et al.  Review of surrogate modeling in water resources , 2012 .

[40]  Diganta Misra Mish: A Self Regularized Non-Monotonic Activation Function , 2020, BMVC.

[41]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Joachim Denzler,et al.  Deep learning and process understanding for data-driven Earth system science , 2019, Nature.

[43]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[44]  Mary F. Wheeler,et al.  Efficient Bayesian inference of subsurface flow models using nested sampling and sparse polynomial chaos surrogates , 2014 .

[45]  Anthony J. Jakeman,et al.  A review of surrogate models and their application to groundwater modeling , 2015 .

[46]  Sebastien Strebelle,et al.  Conditional Simulation of Complex Geological Structures Using Multiple-Point Statistics , 2002 .

[47]  P. Renard,et al.  Geological realism in hydrogeological and geophysical inverse modeling: A review , 2015, 1701.01602.

[48]  Xiang Ma,et al.  Kernel principal component analysis for stochastic input model generation , 2010, J. Comput. Phys..

[49]  Ahmed H. Elsheikh,et al.  Parametrization and generation of geological models with generative adversarial networks , 2017, 1708.01810.

[50]  Liangping Li,et al.  Inverse methods in hydrogeology: Evolution and recent trends , 2014 .

[51]  Zhi Zhong,et al.  Predicting CO2 Plume Migration in Heterogeneous Formations Using Conditional Deep Convolutional Generative Adversarial Network , 2019, Water Resources Research.

[52]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[53]  G. Evensen Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics , 1994 .

[54]  Albert C. Reynolds,et al.  Ensemble smoother with multiple data assimilation , 2013, Comput. Geosci..

[55]  Edgar Thiago de Oliveira Chagas Deep Learning and its Applications today , 2019 .

[56]  P. Kitanidis Persistent questions of heterogeneity, uncertainty, and scale in subsurface flow and transport , 2015 .

[57]  G. Mariéthoz,et al.  Bayesian inverse problem and optimization with iterative spatial resampling , 2010 .

[58]  Lingzao Zeng,et al.  An adaptive Gaussian process‐based method for efficient Bayesian experimental design in groundwater contaminant source identification problems , 2016 .

[59]  Alexander Y. Sun,et al.  Sequential updating of multimodal hydrogeologic parameter fields using localization and clustering techniques , 2009 .

[60]  Eric Laloy,et al.  Training‐Image Based Geostatistical Inversion Using a Spatial Generative Adversarial Neural Network , 2017, ArXiv.

[61]  Kang Chen,et al.  Bridging iterative Ensemble Smoother and multiple-point geostatistics for better flow and transport modeling , 2018, Journal of Hydrology.

[62]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[63]  Guang Lin,et al.  An efficient, high-order probabilistic collocation method on sparse grids for three-dimensional flow and solute transport in randomly heterogeneous porous media , 2009 .

[64]  Eric Laloy,et al.  Gradient-based deterministic inversion of geophysical data with generative adversarial networks: Is it feasible? , 2018, Comput. Geosci..

[65]  Nicholas Zabaras,et al.  Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification , 2018, J. Comput. Phys..

[66]  Liangping Li,et al.  An approach to handling non-Gaussianity of parameters and state variables in ensemble Kalman filtering , 2011 .

[67]  Dilin Wang,et al.  Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm , 2016, NIPS.

[68]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[69]  L. Durlofsky,et al.  Kernel Principal Component Analysis for Efficient, Differentiable Parameterization of Multipoint Geostatistics , 2008 .

[70]  Adam Lipowski,et al.  Roulette-wheel selection via stochastic acceptance , 2011, ArXiv.

[71]  Jichun Wu,et al.  Deep Convolutional Encoder‐Decoder Networks for Uncertainty Quantification of Dynamic Multiphase Flow in Heterogeneous Media , 2018, Water Resources Research.

[72]  Ilias Bilionis,et al.  Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification , 2018, J. Comput. Phys..

[73]  Vincent Dumoulin,et al.  Deconvolution and Checkerboard Artifacts , 2016 .

[74]  Guang Lin,et al.  An Iterative Local Updating Ensemble Smoother for Estimation and Uncertainty Assessment of Hydrologic Model Parameters With Multimodal Distributions , 2016, 1611.04702.

[75]  Dongbin Xiu,et al.  The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations , 2002, SIAM J. Sci. Comput..

[76]  J. Gómez-Hernández,et al.  Simultaneous identification of a contaminant source and hydraulic conductivity via the restart normal-score ensemble Kalman filter , 2018 .

[77]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[78]  Chaopeng Shen,et al.  A Transdisciplinary Review of Deep Learning Research and Its Relevance for Water Resources Scientists , 2017, Water Resources Research.

[79]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[80]  Ahmed H. Elsheikh,et al.  Parametric generation of conditional geological realizations using generative neural networks , 2018, Computational Geosciences.

[81]  Maarten V. de Hoop,et al.  Machine learning for data-driven discovery in solid Earth geoscience , 2019, Science.

[82]  Ahmed H. Elsheikh,et al.  Parametrization of Stochastic Inputs Using Generative Adversarial Networks With Application in Geology , 2019, Frontiers in Water.

[83]  Philippe Renard,et al.  Issues in characterizing heterogeneity and connectivity in non-multiGaussian media , 2008 .

[84]  Yimin Liu,et al.  A Deep-Learning-Based Geological Parameterization for History Matching Complex Models , 2018, Mathematical Geosciences.

[85]  Jiangjiang Zhang,et al.  An adaptive Gaussian process-based iterative ensemble smoother for data assimilation , 2018 .

[86]  Haibin Chang,et al.  Surrogate model based iterative ensemble smoother for subsurface flow data assimilation , 2017 .

[87]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[88]  Alexander Y. Sun,et al.  Discovering State‐Parameter Mappings in Subsurface Models Using Generative Adversarial Networks , 2018, Geophysical Research Letters.

[89]  Paul P. Wang,et al.  MT3DMS: A Modular Three-Dimensional Multispecies Transport Model for Simulation of Advection, Dispersion, and Chemical Reactions of Contaminants in Groundwater Systems; Documentation and User's Guide , 1999 .