Incorporating Unmodeled Dynamics Into First-Principles Models Through Machine Learning

First-principles modeling of dynamical systems is a cornerstone of science and engineering and has enabled rapid development and improvement of key technologies such as chemical reactors, electrical circuits, and communication networks. In various disciplines, scientists structure the available domain knowledge into a system of differential equations. When designed, calibrated, and validated appropriately, these equations are used to analyze and predict the dynamics of the system. However, perfect knowledge is usually not accessible in real-world problems. The incorporated knowledge thus is a simplification of the real system and is limited by the underlying assumptions. This limits the extent to which the model reflects reality. The resulting lack of predictive power severely hampers the application potential of such models. Here we introduce a framework that incorporates machine learning into existing first-principles modeling. The machine learning model fills in the knowledge gaps of the first-principles model, capturing the unmodeled dynamics and thus improving the representativeness of the model. Moreover, we show that this approach lowers the data requirements, both in quantity and quality, and improves the generalization ability in comparison with a purely data-driven approach. This approach can be applied to any first-principles model with sufficient data available and has tremendous potential in many fields.

[1]  Vaibhav Dixit,et al.  A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions , 2018, 2021 IEEE High Performance Extreme Computing Conference (HPEC).

[2]  Sheng Chen,et al.  Non-linear systems identification using radial basis functions , 1990 .

[3]  Tommy W. S. Chow,et al.  Approximation of dynamical time-variant systems by continuous-time recurrent neural networks , 2005, IEEE Transactions on Circuits and Systems II: Express Briefs.

[4]  Hayden Schaeffer,et al.  Extracting Sparse High-Dimensional Dynamics from Limited Data , 2017, SIAM J. Appl. Math..

[5]  Lishan Kang,et al.  Evolutionary Modeling of Systems of Ordinary Differential Equations with Genetic Programming , 2000, Genetic Programming and Evolvable Machines.

[6]  Hod Lipson,et al.  Automated reverse engineering of nonlinear dynamical systems , 2007, Proceedings of the National Academy of Sciences.

[7]  W. K. Lee,et al.  Chaotic Dynamics of a Harmonically Excited Spring-Pendulum System with Internal Resonance , 1997 .

[8]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[9]  Hong Chen,et al.  Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems , 1995, IEEE Trans. Neural Networks.

[10]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[11]  I. Nopens,et al.  Colloidal fouling in electrodialysis: A neural differential equations model , 2020, Separation and Purification Technology.

[12]  Kamran Shafi,et al.  Prediction of dynamical systems by symbolic regression. , 2016, Physical review. E.

[13]  Martin T. Hagan,et al.  Backpropagation Algorithms for a Broad Class of Dynamic Networks , 2007, IEEE Transactions on Neural Networks.

[14]  Agathe Girard,et al.  Dynamic systems identification with Gaussian processes , 2005 .

[15]  Paris Perdikaris,et al.  Machine learning of linear differential equations using Gaussian processes , 2017, J. Comput. Phys..

[16]  A. Zhabotinsky,et al.  Oscillatory clusters in the periodically illuminated, spatially extended Belousov-Zhabotinsky reaction. , 2001, Physical review letters.

[17]  R. J. Field,et al.  Oscillations and Traveling Waves in Chemical Systems , 1985 .

[18]  Michael D. Schmidt,et al.  Automated refinement and inference of analytical models for metabolic networks , 2011, Physical biology.

[19]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[20]  Rick Chartrand,et al.  Numerical Differentiation of Noisy, Nonsmooth Data , 2011 .

[21]  George Sugihara,et al.  Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling , 2015, Proceedings of the National Academy of Sciences.

[22]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[23]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[24]  Rick Chartrand,et al.  Numerical differentiation of noisy, nonsmooth, multidimensional data , 2017, 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[25]  Jie Chen,et al.  Fault diagnosis in nonlinear dynamic systems via neural networks , 1994 .

[26]  Lyle H. Ungar,et al.  A hybrid neural network‐first principles approach to process modeling , 1992 .

[27]  P. Dierckx.  An algorithm for smoothing, differentiation and integration of experimental data using spline functions , 1975 .

[28]  Carol S. Woodward,et al.  Enabling New Flexibility in the SUNDIALS Suite of Nonlinear and Differential/Algebraic Equation Solvers , 2020, ACM Trans. Math. Softw..

[29]  Reinhart Heinrich,et al.  Temperature dependency and temperature compensation in a model of yeast glycolytic oscillations. , 2003, Biophysical chemistry.

[30]  G. Allaire.  A review of adjoint methods for sensitivity analysis, uncertainty quantification and optimization in numerical codes , 2015 .

[31]  Thomas J. McAvoy,et al.  INTEGRATING NEURAL NETWORKS WITH FIRST PRINCIPLES MODELS FOR DYNAMIC MODELING , 1992 .

[32]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[33]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[34]  A. M. Zhabotinskii,et al.  Mechanism and mathematical model of the oscillating bromate-ferroin-bromomalonic acid reaction , 1984 .

[35]  Hod Lipson,et al.  Distilling Free-Form Natural Laws from Experimental Data , 2009, Science.

[36]  S F Masrit,et al.  Structure-unknown non-linear dynamic systems : identification through neural networks , 1991 .

[37]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[38]  G. Karniadakis,et al.  Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems , 2018, 1801.01236.

[39]  M. L. Chambers.  The Mathematical Theory of Optimal Processes , 1965 .

[40]  Bryan C. Daniels,et al.  Efficient Inference of Parsimonious Phenomenological Models of Cellular Dynamics Using S-Systems and Alternating Regression , 2014, PloS one.

[41]  M. Ayoubi.  Nonlinear dynamic systems identification with dynamic neural networks for fault diagnosis in technical processes , 1994, Proceedings of IEEE International Conference on Systems, Man and Cybernetics.

[42]  Rui Oliveira.  Combining first principles modelling and artificial neural networks: a general framework , 2004, Comput. Chem. Eng..

[43]  Karl J. Friston,et al.  Efficient gradient computation for dynamical models , 2014, NeuroImage.

[44]  Ioannis G. Kevrekidis,et al.  Identification of distributed parameter systems: A neural net based approach , 1998 .

[45]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[46]  S. Billings.  Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains , 2013 .