Deep learning for universal linear embeddings of nonlinear dynamics

Identifying coordinate transformations that make strongly nonlinear dynamics approximately linear has the potential to enable nonlinear prediction, estimation, and control using linear theory. The Koopman operator is a leading data-driven embedding, and its eigenfunctions provide intrinsic coordinates that globally linearize the dynamics. However, identifying and representing these eigenfunctions has proven challenging. This work leverages deep learning to discover representations of Koopman eigenfunctions from data. Our network is parsimonious and interpretable by construction, embedding the dynamics on a low-dimensional manifold. We identify nonlinear coordinates on which the dynamics are globally linear using a modified auto-encoder. We also generalize Koopman representations to include a ubiquitous class of systems with continuous spectra. Our framework parametrizes the continuous frequency using an auxiliary network, enabling a compact and efficient embedding, while connecting our models to decades of asymptotics. Thus, we benefit from the power of deep learning, while retaining the physical interpretability of Koopman embeddings.It is often advantageous to transform a strongly nonlinear system into a linear one in order to simplify its analysis for prediction and control. Here the authors combine dynamical systems with deep learning to identify these hard-to-find transformations.

[1]  Hao Wu,et al.  VAMPnets for deep learning of molecular kinetics , 2017, Nature Communications.

[2]  Steven L. Brunton,et al.  Chaos as an intermittently forced linear system , 2016, Nature Communications.

[3]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[4]  Clarence W. Rowley,et al.  A Kernel Approach to Data-Driven Koopman Spectral Analysis , 2014 .

[5]  G. Dullerud,et al.  A Course in Robust Control Theory: A Convex Approach , 2005 .

[6]  Frank Noé,et al.  Variational tensor approach for approximating the rare-event kinetics of macromolecular systems. , 2016, The Journal of chemical physics.

[7]  Steven L. Brunton,et al.  Dynamic mode decomposition - data-driven modeling of complex systems , 2016 .

[8]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[9]  Frank Noé,et al.  Author Correction: VAMPnets for deep learning of molecular kinetics , 2018, Nature Communications.

[10]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[11]  P. Holmes,et al.  Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields , 1983, Applied Mathematical Sciences.

[12]  Michele Milano,et al.  Neural network modeling for near wall turbulent flow , 2002 .

[13]  B. O. Koopman,et al.  Hamiltonian Systems and Transformation in Hilbert Space. , 1931, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Steven L. Brunton,et al.  On dynamic mode decomposition: Theory and applications , 2013, 1312.0041.

[15]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[16]  Hod Lipson,et al.  Automated reverse engineering of nonlinear dynamical systems , 2007, Proceedings of the National Academy of Sciences.

[17]  I. Mezić,et al.  Analysis of Fluid Flows via Spectral Properties of the Koopman Operator , 2013 .

[18]  S. Orszag,et al.  Advanced mathematical methods for scientists and engineers I: asymptotic methods and perturbation theory. , 1999 .

[19]  Hao Wu,et al.  Data-Driven Model Reduction and Transfer Operator Approximation , 2017, J. Nonlinear Sci..

[20]  Naoya Takeishi,et al.  Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition , 2017, NIPS.

[21]  B. Haas,et al.  Corrigendum: Contribution of classical end-joining to PTEN inactivation in p53-mediated glioblastoma formation and drug-resistant survival , 2017, Nature Communications.

[22]  H. Weitzner,et al.  Perturbation Methods in Applied Mathematics , 1969 .

[23]  Clarence W. Rowley,et al.  A Data–Driven Approximation of the Koopman Operator: Extending Dynamic Mode Decomposition , 2014, Journal of Nonlinear Science.

[24]  I. Mezić Spectral Properties of Dynamical Systems, Model Reduction and Decompositions , 2005 .

[25]  M. Cross,et al.  Pattern formation outside of equilibrium , 1993 .

[26]  Igor Mezic,et al.  Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control , 2016, Autom..

[27]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[28]  Stéphane Mallat,et al.  Understanding deep convolutional networks , 2016, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[29]  Igor Mezic,et al.  Koopman Operator Spectrum and Data Analysis , 2017, 1702.07597.

[30]  B. R. Noack,et al.  A hierarchy of low-dimensional models for the transient and post-transient cylinder wake , 2003, Journal of Fluid Mechanics.

[31]  Clarence W. Rowley,et al.  Linearly-Recurrent Autoencoder Networks for Learning Dynamics , 2017, SIAM J. Appl. Dyn. Syst..

[32]  Frank Noé,et al.  A Variational Approach to Modeling Slow Processes in Stochastic Dynamical Systems , 2012, Multiscale Model. Simul..

[33]  P. Schmid,et al.  Dynamic mode decomposition of numerical and experimental data , 2008, Journal of Fluid Mechanics.

[34]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[35]  Kurt Hornik,et al.  Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks , 1990, Neural Networks.

[36]  Anne Nagel,et al.  Perturbation Methods In Applied Mathematics , 2016 .

[37]  Ioannis G Kevrekidis,et al.  Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator. , 2017, Chaos.

[38]  Shervin Bagheri,et al.  Koopman-mode decomposition of the cylinder wake , 2013, Journal of Fluid Mechanics.

[39]  Igor Mezic,et al.  Ergodic Theory, Dynamic Mode Decomposition, and Computation of Spectral Properties of the Koopman Operator , 2016, SIAM J. Appl. Dyn. Syst..

[40]  B. O. Koopman,et al.  Dynamical Systems of Continuous Spectra. , 1932, Proceedings of the National Academy of Sciences of the United States of America.

[41]  Peter Kulchyski and , 2015 .

[42]  Philipp Schlatter,et al.  Spectral analysis of non-linear flows , 2010 .

[43]  Soumya Kundu,et al.  Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems , 2017, 2019 American Control Conference (ACC).

[44]  Ioannis G. Kevrekidis,et al.  Identification of distributed parameter systems: A neural net based approach , 1998 .

[45]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[46]  Igor Mezic,et al.  Spectrum of the Koopman Operator, Spectral Expansions in Functional Spaces, and State-Space Geometry , 2017, Journal of Nonlinear Science.

[47]  Fernando Paganini,et al.  A Course in Robust Control Theory , 2000 .

[48]  Andrzej Banaszuk,et al.  Comparison of systems with complex behavior , 2004 .

[49]  A. Schaft G.E. Dullerud and F. Paganini - A course in robust control theory: a convex approach. Berlin: Springer-Verlag, 2000 (Text in Applied Mathematics ; 36) , 2001 .

[50]  Steven L. Brunton,et al.  Data-driven discovery of Koopman eigenfunctions for control , 2017, Mach. Learn. Sci. Technol..

[51]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[52]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[53]  Steven L. Brunton,et al.  Constrained sparse Galerkin regression , 2016, Journal of Fluid Mechanics.

[54]  Steven L. Brunton,et al.  Koopman Invariant Subspaces and Finite Linear Representations of Nonlinear Dynamical Systems for Control , 2015, PloS one.

[55]  Matthew O. Williams,et al.  A kernel-based method for data-driven koopman spectral analysis , 2016 .

[56]  Hod Lipson,et al.  Distilling Free-Form Natural Laws from Experimental Data , 2009, Science.

[57]  I. Mezić,et al.  Applied Koopmanism. , 2012, Chaos.

[58]  A. Majda,et al.  Nonlinear Laplacian spectral analysis for time series with intermittency and low-frequency variability , 2012, Proceedings of the National Academy of Sciences.

[59]  P. Bahr,et al.  Sampling: Theory and Applications , 2020, Applied and Numerical Harmonic Analysis.

[60]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[61]  I. Mezić,et al.  Spectral analysis of nonlinear flows , 2009, Journal of Fluid Mechanics.

[62]  Frank Noé,et al.  Variational Approach to Molecular Kinetics. , 2014, Journal of chemical theory and computation.

[63]  Frank Noé,et al.  Time-lagged autoencoders: Deep learning of slow collective variables for molecular kinetics , 2017, The Journal of chemical physics.

[64]  Eric P. Xing,et al.  GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server , 2016, EuroSys.

[65]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..