Liquid Time-constant Networks

We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems modulated via nonlinear interlinked gates. The resulting models represent dynamical systems with varying (i.e., \emph{liquid}) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations, and give rise to improved performance on time-series prediction tasks. To demonstrate these properties, we first take a theoretical approach to find bounds over their dynamics, and compute their expressive power by the \emph{trajectory length} measure in a latent trajectory space. We then conduct a series of time-series prediction experiments to manifest the approximation capability of Liquid Time-Constant Networks (LTCs) compared to modern RNNs. Code and data are available at this https URL

[1]  Surya Ganguli,et al.  On the Expressive Power of Deep Neural Networks , 2016, ICML.

[2]  S. R. Wicks,et al.  A Dynamic Network Simulation of the Nematode Tap Withdrawal Circuit: Predictions Concerning Synaptic Function Using Behavioral Criteria , 1996, The Journal of Neuroscience.

[3]  Priscilla Koch Wagner,et al.  Gesture Unit Segmentation Using Spatial-Temporal Information and Machine Learning , 2014, FLAIRS.

[4]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[5]  Kun Zhang,et al.  Forecasting skewed biased stochastic ozone days: analyses, solutions and beyond , 2008, Knowledge and Information Systems.

[6]  J. Duncan,et al.  Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE , 2020, ICML.

[7]  Mathias Lechner,et al.  Learning Long-Term Dependencies in Irregularly-Sampled Time Series , 2020, NeurIPS.

[8]  Ramin M. Hasani.  Interpretable Recurrent Neural Networks in Continuous-time Control Environments , 2020 .

[9]  Karl J. Friston,et al.  Dynamic causal modelling , 2003, NeuroImage.

[10]  J. Lambert.  Computer Solution of Ordinary Differential Equations , 1976 .

[11]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[12]  Lawrence F. Shampine,et al.  Analysis and implementation of TR-BDF2 , 1996 .

[13]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[14]  David Duvenaud,et al.  Latent ODEs for Irregularly-Sampled Time Series , 2019, ArXiv.

[15]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[16]  Radu Grosu,et al.  Neural circuit policies enabling auditable autonomy , 2020, Nature Machine Intelligence.

[17]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[18]  Yee Whye Teh,et al.  Augmented Neural ODEs , 2019, NeurIPS.

[19]  Matthias Durr,et al.  Methods In Neuronal Modeling From Ions To Networks , 2016 .

[20]  L. Shampine,et al.  Computer solution of ordinary differential equations : the initial value problem , 1975 .

[21]  Razvan Pascanu,et al.  On the number of response regions of deep feed forward networks with piece-wise linear activations , 2013, 1312.6098.

[22]  Surya Ganguli,et al.  Exponential expressivity in deep neural networks through transient chaos , 2016, NIPS.

[23]  Ohad Shamir,et al.  The Power of Depth for Feedforward Neural Networks , 2015, COLT.

[24]  William H. Press,et al.  Numerical Recipes 3rd Edition: The Art of Scientific Computing , 2007 .

[25]  Radu Grosu,et al.  Designing Worm-inspired Neural Networks for Interpretable Robotic Control , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[26]  Hossein Hassani,et al.  On the Folded Normal Distribution , 2014, 1402.3559.

[27]  Austin R. Benson,et al.  Neural Jump Stochastic Differential Equations , 2019, NeurIPS.

[28]  L. Shampine,et al.  A 3(2) pair of Runge - Kutta formulas , 1989 .

[29]  Christian Tjandraatmadja,et al.  Bounding and Counting Linear Regions of Deep Neural Networks , 2017, ICML.

[30]  David Rolnick,et al.  Complexity of Linear Regions in Deep Networks , 2019, ICML.

[31]  Nicolas Macris,et al.  Entropy and mutual information in models of deep neural networks , 2018, NeurIPS.

[32]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[33]  Razvan Pascanu,et al.  On the Number of Linear Regions of Deep Neural Networks , 2014, NIPS.

[34]  Tommi S. Jaakkola,et al.  Towards Robust, Locally Linear Deep Networks , 2019, ICLR.

[35]  Karl J. Friston,et al.  Bilinear dynamical systems , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[36]  Radu Grosu,et al.  Non-Associative Learning Representation in the Nervous System of the Nematode Caenorhabditis elegans , 2017, ArXiv.

[37]  Vincent Y. F. Tan,et al.  On Robustness of Neural Ordinary Differential Equations , 2020, ICLR.

[38]  M. L. Chambers.  The Mathematical Theory of Optimal Processes , 1965 .

[39]  Kurt Keutzer,et al.  ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs , 2019, IJCAI.

[40]  Hans-Georg Zimmermann,et al.  Recurrent Neural Networks Are Universal Approximators , 2006, ICANN.

[41]  Michael C. Mozer,et al.  Discrete Event, Continuous Time RNNs , 2017, ArXiv.

[42]  Bernhard Scholkopf.  Causality for Machine Learning , 2019 .

[43]  Richard Gordon,et al.  OpenWorm: overview and recent advances in integrative biological simulation of Caenorhabditis elegans , 2018, Philosophical Transactions of the Royal Society B.

[44]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[45]  Yuval Tassa,et al.  MuJoCo: A physics engine for model-based control , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[46]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[47]  Radu Grosu,et al.  c302: a multiscale framework for modelling the nervous system of Caenorhabditis elegans , 2018, Philosophical Transactions of the Royal Society B: Biological Sciences.

[48]  J. Dormand,et al.  A family of embedded Runge-Kutta formulae , 1980 .

[49]  Vladlen Koltun,et al.  Learning to Control PDEs with Differentiable Physics , 2020, ICLR.

[50]  Radu Grosu,et al.  Gershgorin Loss Stabilizes the Recurrent Neural Network Compartment of an End-to-end Robot Learning Scheme , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[51]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[52]  Radu Grosu,et al.  A Natural Lottery Ticket Winner: Reinforcement Learning with Ordinary Neural Circuits , 2020, ICML.

[53]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[54]  D. Arnold,et al.  Computer Solution of Ordinary Differential Equations. , 1981 .

[55]  Radu Grosu,et al.  Response Characterization for Auditing Cell Dynamics in Long Short-term Memory Networks , 2018, 2019 International Joint Conference on Neural Networks (IJCNN).

[56]  David Rolnick,et al.  How to Start Training: The Effect of Initialization and Architecture , 2018, NeurIPS.

[57]  Joseph DelPreto,et al.  Plug-and-play supervisory control using muscle and brain signals for real-time gesture and error detection , 2018, Autonomous Robots.

[58]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[59]  Iain Murray,et al.  Neural Spline Flows , 2019, NeurIPS.

[60]  M. Hirsch,et al.  Differential Equations, Dynamical Systems, and Linear Algebra , 1974 .

[61]  Luis M. Candanedo,et al.  Accurate occupancy detection of an office room from light, temperature, humidity and CO2 measurements using statistical learning models , 2016 .

[62]  Davide Anguita,et al.  A Public Domain Dataset for Human Activity Recognition using Smartphones , 2013, ESANN.