On the application of Physically-Guided Neural Networks with Internal Variables to Continuum Problems

Predictive Physics has been historically based upon the development of mathematical models that describe the evolution of a system under certain external stimuli and constraints. The structure of such mathematical models relies on a set of hysical hypotheses that are assumed to be fulfilled by the system within a certain range of environmental conditions. A new perspective is now raising that uses physical knowledge to inform the data prediction capability of artificial neural networks. A particular extension of this data-driven approach is Physically-Guided Neural Networks with Internal Variables (PGNNIV): universal physical laws are used as constraints in the neural network, in such a way that some neuron values can be interpreted as internal state variables of the system. This endows the network with unraveling capacity, as well as better predictive properties such as faster convergence, fewer data needs and additional noise filtering. Besides, only observable data are used to train the network, and the internal state equations may be extracted as a result of the training processes, so there is no need to make explicit the particular structure of the internal state model. We extend this new methodology to continuum physical problems, showing again its predictive and explanatory capacities when only using measurable values in the training set. We show that the mathematical operators developed for image analysis in deep learning approaches can be used and extended to consider standard functional operators in continuum Physics, thus establishing a common framework for both. The methodology presented demonstrates its ability to discover the internal constitutive state equation for some problems, including heterogeneous and nonlinear features, while maintaining its predictive ability for the whole dataset coverage, with the cost of a single evaluation.

[1]  Phillipp Meister,et al.  Theory Of Fluid Flows Through Natural Rocks , 2016 .

[2]  Liwei Wang,et al.  The Expressive Power of Neural Networks: A View from the Width , 2017, NIPS.

[3]  Matthias Ehrhardt,et al.  On the numerical solution of nonlinear Black-Scholes equations , 2008, Comput. Math. Appl..

[4]  J. L. Varona,et al.  Nonlocal discrete diffusion equations and the fractional discrete Laplacian, regularity and applications , 2016, 1608.08913.

[5]  Xavier Ros-Oton,et al.  Nonlocal elliptic equations in bounded domains: a survey , 2015, 1504.04099.

[6]  J. Monaghan,et al.  Smoothed particle hydrodynamics: Theory and application to non-spherical stars , 1977 .

[7]  Manuel Doblaré,et al.  A new reliability-based data-driven approach for noisy experimental data with physical constraints , 2018 .

[8]  Paris Perdikaris,et al.  Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations , 2017, ArXiv.

[9]  Peter Gould,et al.  LETTING THE DATA SPEAK FOR THEMSELVES , 1981 .

[10]  Olivier Sigaud,et al.  Many regression algorithms, one unified model: A review , 2015, Neural Networks.

[11]  Boris Hanin,et al.  Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations , 2017, Mathematics.

[12]  Alexander Schwartz Logic Inductive And Deductive , 2016 .

[13]  Lorenzo Rosasco,et al.  Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review , 2016, International Journal of Automation and Computing.

[14]  Syed Muhammad Anwar,et al.  Medical Image Analysis using Convolutional Neural Networks: A Review , 2017, Journal of Medical Systems.

[15]  J. Manyika Big data: The next frontier for innovation, competition, and productivity , 2011 .

[16]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[17]  Razvan Pascanu,et al.  Theano: new features and speed improvements , 2012, ArXiv.

[18]  Razvan Pascanu,et al.  Theano: Deep Learning on GPUs with Python , 2012 .

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  Kenneth Levenberg A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[21]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[22]  Cameron D. Palmer,et al.  Association Testing of Previously Reported Variants in a Large Case-Control Meta-analysis of Diabetic Nephropathy , 2011, Diabetes.

[23]  Hans Petter Langtangen,et al.  Computational Partial Differential Equations - Numerical Methods and Diffpack Programming , 1999, Lecture Notes in Computational Science and Engineering.

[24]  Tara N. Sainath,et al.  Deep Learning for Audio Signal Processing , 2019, IEEE Journal of Selected Topics in Signal Processing.

[25]  Yann LeCun,et al.  1.1 Deep Learning Hardware: Past, Present, and Future , 2019, 2019 IEEE International Solid- State Circuits Conference - (ISSCC).

[26]  F. Hirata,et al.  An integral equation theory for inhomogeneous molecular fluids: the reference interaction site model approach. , 2008, The Journal of chemical physics.

[27]  J. Vázquez,et al.  Nonlinear Porous Medium Flow with Fractional Potential Pressure , 2010, 1001.0410.

[28]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[29]  Anuj Karpatne,et al.  Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling , 2017, ArXiv.

[30]  T. Frank Nonlinear Fokker-Planck Equations: Fundamentals and Applications , 2004 .

[31]  Michael Unser,et al.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review , 2017, IEEE Signal Processing Magazine.

[32]  Leah Bar,et al.  Unsupervised Deep Learning Algorithm for PDE-based Forward and Inverse Problems , 2019, ArXiv.

[33]  Chris Volinsky,et al.  Network-Based Marketing: Identifying Likely Adopters Via Consumer Networks , 2006, math/0606278.

[34]  Paris Perdikaris,et al.  Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations , 2017, ArXiv.

[35]  Ruben Juanes,et al.  A deep learning framework for solution and discovery in solid mechanics , 2020 .

[36]  Julia S. Mullen,et al.  Filter-based stabilization of spectral element methods , 2001 .

[37]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[38]  Frederica Darema,et al.  Dynamic Data Driven Applications Systems: A New Paradigm for Application Simulations and Measurements , 2004, International Conference on Computational Science.

[39]  O. Zienkiewicz The Finite Element Method In Engineering Science , 1971 .

[40]  Hyeon-Joong Yoo,et al.  Deep Convolution Neural Networks in Computer Vision: a Review , 2015 .

[41]  Ziming Yan,et al.  Combination and application of machine learning and computational mechanics , 2019, Chinese Science Bulletin.

[42]  Zhiping Mao,et al.  DeepXDE: A Deep Learning Library for Solving Differential Equations , 2019, AAAI Spring Symposium: MLPS.

[43]  Guy Barles,et al.  Option pricing with transaction costs and a nonlinear Black-Scholes equation , 1998, Finance Stochastics.

[44]  B. Nayroles,et al.  Generalizing the finite element method: Diffuse approximation and diffuse elements , 1992 .

[45]  Joaquín González-Rodríguez,et al.  Automatic language identification using deep neural networks , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[46]  Philippe Lorong,et al.  Natural Element Method for the Simulation of Structures and Processes: Chinesta/Natural Element Method for the Simulation of Structures and Processes , 2011 .

[47]  Gerald M. Maggiora,et al.  Computational neural networks as model-free mapping devices , 1992, J. Chem. Inf. Comput. Sci..

[48]  J. A. Sanz-Herrera,et al.  Identification of state functions by physically-guided neural networks with physically-meaningful internal layers , 2020, ArXiv.

[49]  Lida Xu,et al.  The internet of things: a survey , 2014, Information Systems Frontiers.

[50]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[51]  Numerical methods for partial differential equations , 2016 .

[52]  R. Osserman A survey of minimal surfaces , 1969 .

[53]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[54]  Joel H. Saltz,et al.  ConvNets with Smooth Adaptive Activation Functions for Regression , 2017, AISTATS.

[55]  Antonio J. Gil,et al.  Nonlinear Solid Mechanics for Finite Element Analysis: Statics , 2016 .

[56]  Aurélien Géron,et al.  Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems , 2017 .

[57]  Bin Dong,et al.  PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network , 2018, J. Comput. Phys..

[58]  Nagiza F. Samatova,et al.  Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data , 2016, IEEE Transactions on Knowledge and Data Engineering.

[59]  Manuel Doblaré,et al.  An unsupervised data completion method for physically-based data-driven models , 2019, Computer Methods in Applied Mechanics and Engineering.

[60]  T. Belytschko,et al.  THE NATURAL ELEMENT METHOD IN SOLID MECHANICS , 1998 .

[61]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[62]  Stig Larsson,et al.  Partial differential equations with numerical methods , 2003, Texts in applied mathematics.

[63]  Dongyan Zhao,et al.  Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges , 2019, NLPCC.

[64]  Trenton Kirchdoerfer,et al.  Data-driven computational mechanics , 2015, 1510.04232.

[65]  Markus Reischl,et al.  Benchmarking in classification and regression , 2019, Wiley Interdiscip. Rev. Data Min. Knowl. Discov..

[66]  Benjamin Peherstorfer,et al.  Dynamic data-driven reduced-order models , 2015 .

[67]  Steven L. Brunton,et al.  Dynamic mode decomposition - data-driven modeling of complex systems , 2016 .

[68]  R. Kitchin,et al.  Big Data, new epistemologies and paradigm shifts , 2014, Big Data Soc..

[69]  Xue Ying,et al.  An Overview of Overfitting and its Solutions , 2019, Journal of Physics: Conference Series.

[70]  Zenghui Wang,et al.  Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review , 2017, Neural Computation.