Prediction and identification of physical systems by means of Physically-Guided Neural Networks with meaningful internal layers

Abstract Substitution of well-grounded theoretical models by data-driven predictions is not as simple in engineering and sciences as it is in social and economic fields. Scientific problems suffer many times from paucity of data, while they may involve a large number of variables and parameters that interact in complex and non-stationary ways, obeying certain physical laws. Moreover, a physically-based model is not only useful for making predictions, but to gain knowledge by the interpretation of its structure, parameters, and mathematical properties . The solution to these shortcomings seems to be the seamless blending of the tremendous predictive power of the data-driven approach with the scientific consistency and interpretability of physically-based models. We use here the concept of Physically-Guided Neural Networks (PGNN) to predict the input–output relation in a physical system, while, at the same time, fulfilling the physical constraints. With this goal, the internal hidden state variables of the system are associated with a set of internal neuron layers, whose values are constrained by known physical relations, as well as any additional knowledge on the system. Furthermore, when having enough data, it is possible to infer knowledge about the internal structure of the system and, if parameterized, to predict the state parameters for a particular input–output relation. We show that this approach, besides getting physically-based predictions, accelerates the training process, reduces the amount of data required to get similar accuracy, partly filters the intrinsic noise in the experimental data and improves its extrapolation capacity.

[1]  José M. Matías,et al.  Boosting GARCH and neural networks for the prediction of heteroskedastic time series , 2010, Math. Comput. Model..

[2]  Ruey‐Shiang Guh.  Robustness of the neural network based control chart pattern recognition system to non‐normality , 2002 .

[3]  Clive W. J. Granger,et al.  Testing for neglected nonlinearity in time series models: A comparison of neural network methods and alternative tests , 1993 .

[4]  Ridha Hambli,et al.  Real-time deformation of structure using finite element and neural networks in virtual reality applications , 2006 .

[5]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[6]  George Em Karniadakis,et al.  Systems biology informed deep learning for inferring parameters and hidden dynamics , 2020, PLoS computational biology.

[7]  Viju Raghupathi,et al.  Big data analytics in healthcare: promise and potential , 2014, Health Information Science and Systems.

[8]  Davide Castelvecchi,et al.  Can we open the black box of AI? , 2016, Nature.

[9]  Frederica Darema,et al.  Dynamic Data Driven Applications Systems: A New Paradigm for Application Simulations and Measurements , 2004, International Conference on Computational Science.

[10]  George Em Karniadakis,et al.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations , 2020, Science.

[11]  David J. J. Toal,et al.  Kriging Hyperparameter Tuning Strategies , 2008 .

[12]  Cameron D. Palmer,et al.  Association Testing of Previously Reported Variants in a Large Case-Control Meta-analysis of Diabetic Nephropathy , 2011, Diabetes.

[13]  Klaus Kofler,et al.  Performance and Scalability of GPU-Based Convolutional Neural Networks , 2010, 2010 18th Euromicro Conference on Parallel, Distributed and Network-based Processing.

[14]  Francisco J. Montáns,et al.  What-You-Prescribe-Is-What-You-Get orthotropic hyperelasticity , 2014 .

[15]  Zhiping Mao,et al.  DeepXDE: A Deep Learning Library for Solving Differential Equations , 2019, AAAI Spring Symposium: MLPS.

[16]  E. Lopez,et al.  A Manifold Learning Approach for Integrated Computational Materials Engineering , 2017 .

[17]  G. Karniadakis,et al.  Physics-informed neural networks for high-speed flows , 2020, Computer Methods in Applied Mechanics and Engineering.

[18]  Michael McAleer,et al.  A neural network demand system with heteroskedastic errors , 2008 .

[19]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[20]  David González,et al.  Thermodynamically consistent data-driven computational mechanics , 2018, Continuum Mechanics and Thermodynamics.

[21]  Ananthram Swami,et al.  Practical Black-Box Attacks against Machine Learning , 2016, AsiaCCS.

[22]  Hod Lipson,et al.  Distilling Free-Form Natural Laws from Experimental Data , 2009, Science.

[23]  Joaquín González-Rodríguez,et al.  Automatic language identification using deep neural networks , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[24]  Filip Karlo Dosilovic,et al.  Explainable artificial intelligence: A survey , 2018, 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[25]  Nir Shavit,et al.  Deep Learning is Robust to Massive Label Noise , 2017, ArXiv.

[26]  Robert P. W. Duin,et al.  K-nearest Neighbors Directed Noise Injection in Multilayer Perceptron Training , 2000, IEEE Trans. Neural Networks Learn. Syst..

[27]  Naftali Tishby,et al.  Opening the Black Box of Deep Neural Networks via Information , 2017, ArXiv.

[28]  J. Manyika.  Big data: The next frontier for innovation, competition, and productivity , 2011 .

[29]  Anuj Karpatne,et al.  Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling , 2017, ArXiv.

[30]  Chris Volinsky,et al.  Network-Based Marketing: Identifying Likely Adopters Via Consumer Networks , 2006, math/0606278.

[31]  Yves Grandvalet,et al.  Noise Injection: Theoretical Prospects , 1997, Neural Computation.

[32]  Peter Gould,et al.  LETTING THE DATA SPEAK FOR THEMSELVES , 1981 .

[33]  C. Lee Giles,et al.  An analysis of noise in recurrent neural networks: convergence and generalization , 1996, IEEE Trans. Neural Networks.

[34]  Benjamin Peherstorfer,et al.  Dynamic data-driven reduced-order models , 2015 .

[35]  Julien Yvonnet,et al.  A data-driven computational homogenization method based on neural networks for the nonlinear anisotropic electrical response of graphene/polymer nanocomposites , 2018, Computational Mechanics.

[36]  Trenton Kirchdoerfer,et al.  Data-driven computational mechanics , 2015, 1510.04232.

[37]  Manuel Doblaré,et al.  An unsupervised data completion method for physically-based data-driven models , 2019, Computer Methods in Applied Mechanics and Engineering.

[38]  Ryan P. Adams,et al.  Gradient-based Hyperparameter Optimization through Reversible Learning , 2015, ICML.

[39]  Petri Koistinen,et al.  Using additive noise in back-propagation training , 1992, IEEE Trans. Neural Networks.

[40]  Liwei Wang,et al.  The Expressive Power of Neural Networks: A View from the Width , 2017, NIPS.

[41]  Antonio Iera,et al.  The Internet of Things: A survey , 2010, Comput. Networks.

[42]  Víctor Jesús Amores,et al.  Average-chain behavior of isotropic incompressible polymers obtained from macroscopic experimental data. A simple structure-based WYPiWYG model in Julia language , 2019, Adv. Eng. Softw..

[43]  David González,et al.  Learning Corrections for Hyperelastic Models From Data , 2019, Front. Mater..

[44]  Jules Thibault,et al.  A neural network methodology for heat transfer data analysis , 1991 .

[45]  K. K. Pathak,et al.  Application of Neural Network in Sheet Metal Bending Process , 2005 .

[46]  Maziar Raissi,et al.  Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations , 2018, J. Mach. Learn. Res..

[47]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[48]  R. Kitchin,et al.  Big Data, new epistemologies and paradigm shifts , 2014, Big Data Soc..

[49]  E. Cueto,et al.  Learning slosh dynamics by means of data , 2019, Computational Mechanics.

[50]  Michèle Sebag,et al.  Collaborative hyperparameter tuning , 2013, ICML.

[51]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[52]  Benjamin Skuse.  The third pillar , 2019 .

[53]  Klaus-Jürgen Bathe,et al.  A model of incompressible isotropic hyperelastic material behavior using spline interpolations of tension–compression test data , 2009 .

[54]  Francisco Chinesta,et al.  A Manifold Learning Approach to Data-Driven Computational Elasticity and Inelasticity , 2016, Archives of Computational Methods in Engineering.

[55]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[56]  S. Leonelli,et al.  Introduction: Making sense of data-driven research in the biological and biomedical sciences. , 2012, Studies in history and philosophy of biological and biomedical sciences.

[57]  Klaus-Robert Müller,et al.  Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models , 2017, ArXiv.

[58]  Boris Hanin,et al.  Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations , 2017, Mathematics.

[59]  George Keith Batchelor,et al.  An Introduction to Fluid Dynamics. , 1969 .

[60]  Patrick J. Fox,et al.  Shake Table Test of Large-Scale Bridge Columns Supported on Rocking Shallow Foundations , 2015 .

[61]  Nagiza F. Samatova,et al.  Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data , 2016, IEEE Transactions on Knowledge and Data Engineering.