Stochastic deep collocation method based on neural architecture search and transfer learning for heterogeneous porous media

In this work, a modified neural architecture search method (NAS) based physics-informed deep learning model is presented for stochastic analysis in heterogeneous porous material. Monte Carlo method based on a randomized spectral representation is first employed to construct a stochastic model for simulation of flow through porous media. To solve the governing equations for stochastic groundwater flow problem, we build a modified NAS model based on physics-informed neural networks (PINNs) with transfer learning in this paper that will be able to fit different partial differential equations (PDEs) with less calculation. The performance estimation strategies adopted is constructed from an error estimation model using the method of manufactured solutions. A sensitivity analysis is performed to obtain the prior knowledge of the PINNs model and narrow down the range of parameters for search space and use hyper-parameter optimization algorithms to further determine the values of the parameters. Further the NAS based PINNs model also saves the weights and biases of the most favorable architectures, then used in the fine-tuning process. It is found that the log-conductivity field using Gaussian correlation function will perform much better than exponential correlation case, which is more fitted to the PINNs model and the modified neural architecture search based PINNs model shows a great potential in approximating solutions to PDEs. Moreover, a three dimensional stochastic flow model is built to provide a benchmark to the simulation of groundwater flow in highly heterogeneous aquifers. The NAS model based deep collocation method is verified to be effective and accurate through numerical examples in different dimensions using different manufactured solutions.

[1]  Khaled Shaalan,et al.  Speech Recognition Using Deep Neural Networks: A Systematic Review , 2019, IEEE Access.

[2]  Virgilio Fiorotto,et al.  Solute transport in highly heterogeneous aquifers , 1998 .

[3]  Malcolm R Leadbetter,et al.  Stationary and Related Stochastic Processes: Sample Function Properties and Their Applications , 1967 .

[4]  Maziar Raissi,et al.  Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations , 2018, J. Mach. Learn. Res..

[5]  T. Rabczuk,et al.  A Deep Collocation Method for the Bending Analysis of Kirchhoff Plate , 2021, Computers, Materials & Continua.

[6]  Stefano Tarantola,et al.  Uncertainty and sensitivity analysis: tools for GIS-based model implementation , 2001, Int. J. Geogr. Inf. Sci..

[7]  R. Freeze.  A stochastic‐conceptual analysis of one‐dimensional groundwater flow in nonuniform homogeneous media , 1975 .

[8]  Will Usher,et al.  SALib: An open-source Python library for Sensitivity Analysis , 2017, J. Open Source Softw..

[9]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[10]  Willy Bauwens,et al.  Sobol' sensitivity analysis of a complex environmental model , 2011, Environ. Model. Softw..

[11]  A. Mantoglou,et al.  The Turning Bands Method for simulation of random fields using line generation by a spectral method , 1982 .

[12]  Zhonghua Qiao,et al.  Numerical Solution of Differential Equations: Introduction to Finite Difference and Finite Element Methods , 2017 .

[13]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[14]  Timon Rabczuk,et al.  Transfer learning enhanced physics informed neural network for phase-field modeling of fracture , 2019, Theoretical and Applied Fracture Mechanics.

[15]  Frank Hutter,et al.  Neural Architecture Search: A Survey , 2018, J. Mach. Learn. Res..

[16]  Xiaoying Zhuang,et al.  A deep energy method for finite deformation hyperelasticity , 2020 .

[17]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[18]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[19]  G. Dagan.  Flow and transport in porous formations , 1989 .

[20]  Aaron Klein,et al.  Efficient and Robust Automated Machine Learning , 2015, NIPS.

[21]  Stephane Etienne,et al.  Code Verification and the Method of Manufactured Solutions for Fluid-Structure Interaction Problems , 2006 .

[22]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[23]  E Weinan,et al.  Machine Learning Approximation Algorithms for High-Dimensional Fully Nonlinear Partial Differential Equations and Second-order Backward Stochastic Differential Equations , 2017, J. Nonlinear Sci..

[24]  Quoc V. Le,et al.  Efficient Neural Architecture Search via Parameter Sharing , 2018, ICML.

[25]  Jingye Li,et al.  Physics informed neural networks for velocity inversion , 2019, SEG Technical Program Expanded Abstracts 2019.

[26]  Su Wei,et al.  Sensitivity analysis of CERES-Wheat model parameters based on EFAST method. , 2012 .

[27]  R. Ababou,et al.  Numerical simulation of three-dimensional saturated flow in randomly heterogeneous porous media , 1989 .

[28]  Karl W. Schulz,et al.  MASA: a library for verification using manufactured and analytical solutions , 2012, Engineering with Computers.

[29]  Brian Henderson-Sellers,et al.  Sensitivity evaluation of environmental models using fractional factorial experimentation , 1996 .

[30]  Ameet Talwalkar,et al.  Non-stochastic Best Arm Identification and Hyperparameter Optimization , 2015, AISTATS.

[31]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[32]  H. Rumpf,et al.  The influence of porosity and grain size distribution on the permeability equation of porous flow , 1975 .

[33]  Sabine Attinger,et al.  Generalized Coarse Graining Procedures for Flow in Porous Media , 2003 .

[34]  Naif Alajlan,et al.  Artificial Neural Network Methods for the Solution of Second Order Boundary Value Problems , 2019, Computers, Materials & Continua.

[35]  Xindong Wu,et al.  Object Detection With Deep Learning: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[36]  Lynn W. Gelhar,et al.  Stochastic subsurface hydrology from theory to applications , 1986 .

[37]  Allan L. Gutjahr,et al.  Stochastic analysis of spatial variability in subsurface flows: 1. Comparison of one‐ and three‐dimensional flows , 1978 .

[38]  C. Axness,et al.  Three‐dimensional stochastic analysis of macrodispersion in aquifers , 1983 .

[39]  Karl K. Sabelfeld,et al.  Stochastic Flow Simulation and Particle Transport in a 2D Layer of Random Porous Medium , 2010 .

[40]  Liping Yang,et al.  Visually-Enabled Active Deep Learning for (Geo) Text and Image Classification: A Review , 2018, ISPRS Int. J. Geo Inf..

[41]  Haohan Wang,et al.  Deep Learning for Genomics: A Concise Overview , 2018, ArXiv.

[42]  Timon Rabczuk,et al.  An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications , 2019, Computer Methods in Applied Mechanics and Engineering.

[43]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[44]  George E. Karniadakis,et al.  Hidden physics models: Machine learning of nonlinear partial differential equations , 2017, J. Comput. Phys..

[45]  Trang T. Le,et al.  Scaling tree-based automated machine learning to biomedical big data with a feature set selector , 2019, Bioinform..

[46]  Sebastian Becker,et al.  Solving stochastic differential equations and Kolmogorov equations by means of deep learning , 2018, ArXiv.

[47]  Spyros Chatzivasileiadis,et al.  Physics-Informed Neural Networks for Power Systems , 2020, 2020 IEEE Power & Energy Society General Meeting (PESGM).

[48]  H. Pape,et al.  Variation of Permeability with Porosity in Sandstone Diagenesis Interpreted with a Fractal Pore Space Model , 2000 .

[49]  G. Matheron,et al.  Is transport in porous media always diffusive? A counterexample , 1980 .

[50]  Quoc V. Le,et al.  Neural Architecture Search with Reinforcement Learning , 2016, ICLR.

[51]  Anqi Wang,et al.  Practical Experience of Sensitivity Analysis: Comparing Six Methods, on Three Hydrological Models, with Three Performance Criteria , 2019, Water.

[52]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[53]  Thomas Fischer,et al.  Deep learning with long short-term memory networks for financial market predictions , 2017, Eur. J. Oper. Res..

[54]  Peter R. Kramer,et al.  Comparative analysis of multiscale Gaussian random field simulation algorithms , 2007, J. Comput. Phys..

[55]  Kyle Mills,et al.  Deep learning and the Schrödinger equation , 2017, ArXiv.

[56]  Ali Al-Aradi,et al.  Solving Nonlinear and High-Dimensional Partial Differential Equations via Deep Learning , 2018, 1811.08782.

[57]  Ameet Talwalkar,et al.  Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization , 2016, J. Mach. Learn. Res..

[58]  E Weinan,et al.  Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations , 2017, Communications in Mathematics and Statistics.

[59]  Arnulf Jentzen,et al.  Solving high-dimensional partial differential equations using deep learning , 2017, Proceedings of the National Academy of Sciences.

[60]  Frank Gauterin,et al.  Dimensionality reduction and identification of valid parameter bounds for the efficient calibration of automated driving functions , 2019 .

[61]  A. Saltelli,et al.  A quantitative model-independent method for global sensitivity analysis of model output , 1999 .

[62]  Walter A. Illman,et al.  Heterogeneity in hydraulic conductivity and its role on the macroscale transport of a solute plume: From measurements to a practical application of stochastic flow and transport theory , 2010 .

[63]  Jacek Majkowski,et al.  Multiplicative sensitivity analysis and its role in development of simulation models , 1981 .

[64]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[65]  Yoram Rubin,et al.  HYDRO_GEN: A spatially distributed random field generator for correlated properties , 1996 .

[66]  Rodolphe Le Riche,et al.  Comparison of different global sensitivity analysis methods for aerospace vehicle optimal design , 2013 .

[67]  Amir Mosavi,et al.  Deep Learning: A Review , 2018 .

[68]  Josh Patterson,et al.  Deep Learning: A Practitioner's Approach , 2017 .

[69]  Maziar Raissi,et al.  Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations , 2018, ArXiv.

[70]  P. Carman.  Fluid flow through granular beds , 1997 .

[71]  Robert V. O'Neill,et al.  A comparison of sensitivity analysis and error analysis based on a stream ecosystem model , 1981 .

[72]  Li Fei-Fei,et al.  Progressive Neural Architecture Search , 2017, ECCV.

[73]  Karl Sabelfeld,et al.  Stochastic flow simulation in 3D porous media , 2005, Monte Carlo Methods Appl..

[74]  R. Kraichnan.  Diffusion by a Random Velocity Field , 1970 .

[75]  John H. Seinfeld,et al.  Global sensitivity analysis—a computational implementation of the Fourier Amplitude Sensitivity Test (FAST) , 1982 .

[76]  Peter Knabner,et al.  Benchmark for numerical solutions of flow in heterogeneous groundwater formations , 2019, Advances in Water Resources.