Randomized Signature Layers for Signal Extraction in Time Series Data

Time series analysis is a widespread task in Natural Sciences, Social Sciences, and Engineering. A fundamental problem is finding an expressive yet efficient-tocompute representation of the input time series to use as a starting point to perform arbitrary downstream tasks. In this paper, we build upon recent works that use the Signature of a path as a feature map and investigate a computationally efficient technique to approximate these features based on linear random projections. We present several theoretical results to justify our approach and empirically validate that our random projections can effectively retrieve the underlying Signature of a path. We show the surprising performance of the proposed random features on several tasks, including (1) mapping the controls of stochastic differential equations to the corresponding solutions and (2) using the Randomized Signatures as time series representation for classification tasks.When compared to corresponding truncated Signature approaches, our Randomizes Signatures are more computationally efficient in high dimensions, and often lead to better accuracy and faster training. Besides providing a new tool to extract Signatures and further validating the high level of expressiveness of such features, we believe our results provide interesting conceptual links between several existing research areas, suggesting new intriguing directions for future investigations.

[1]  Matthew Daigle,et al.  Electrochemistry-based Battery Modeling for Prognostics , 2013 .

[2]  Yann LeCun,et al.  Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks , 2018, ArXiv.

[3]  Alexandros Karatzoglou,et al.  Towards a universal neural network encoder for time series , 2018, CCIA.

[4]  Patrick Kidger,et al.  Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU , 2020, ICLR.

[5]  B. Øksendal,et al.  Stochastic Calculus for Fractional Brownian Motion and Applications , 2008 .

[6]  James Morrill,et al.  Neural Controlled Differential Equations for Online Prediction Tasks , 2021, ArXiv.

[7]  Benjamin Schrauwen,et al.  An overview of reservoir computing: theory, applications and implementations , 2007, ESANN.

[8]  Alberto Bemporad,et al.  Learning Nonlinear State-Space Models Using Deep Autoencoders , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[9]  John Cristian Borges Gamboa,et al.  Deep Learning for Time-Series Analysis , 2017, ArXiv.

[10]  Terry Lyons,et al.  Uniqueness for the signature of a path of bounded variation and the reduced path group , 2005, math/0507536.

[11]  Gary Marcus,et al.  Deep Learning: A Critical Appraisal , 2018, ArXiv.

[12]  Terry Lyons.  Rough paths, Signatures and the modelling of functions on streams , 2014, 1405.4537.

[13]  Erik Bollt,et al.  Next generation reservoir computing , 2021, Nature Communications.

[14]  Lukas Gonon,et al.  Discrete-Time Signatures and Randomness in Reservoir Computing , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Germain Forestier,et al.  Deep learning for time series classification: a review , 2018, Data Mining and Knowledge Discovery.

[16]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[17]  Brian Ingalls,et al.  Mathematical Modeling in Systems Biology: An Introduction , 2013 .

[18]  Yi Zheng,et al.  Time Series Classification Using Multi-Channels Deep Convolutional Neural Networks , 2014, WAIM.

[19]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[20]  Romain Tavenard,et al.  Data Augmentation for Time Series Classification using Convolutional Neural Networks , 2016 .

[21]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[22]  Terry Lyons,et al.  Neural Controlled Differential Equations for Irregular Time Series , 2020, NeurIPS.

[23]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[24]  Benjamin Recht,et al.  Random Features for Large-Scale Kernel Machines , 2007, NIPS.

[25]  École d'été de probabilités de Saint-Flour,et al.  Differential equations driven by rough paths , 2007 .

[26]  Eamonn J. Keogh,et al.  The UCR time series archive , 2018, IEEE/CAA Journal of Automatica Sinica.

[27]  Pattreeya Tanisaro,et al.  Time Series Classification Using Time Warping Invariant Echo State Networks , 2016, 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA).

[28]  Paolo Magni,et al.  Predictive Pharmacokinetic-Pharmacodynamic Modeling of Tumor Growth Kinetics in Xenograft Models after Administration of Anticancer Agents , 2004, Cancer Research.

[29]  Tim Oates,et al.  Time series classification from scratch with deep neural networks: A strong baseline , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[30]  Lennart Ljung,et al.  Nonlinear System Identification: A User-Oriented Road Map , 2019, IEEE Control Systems.

[31]  Patrick Kidger,et al.  Deep Signatures , 2019, NeurIPS 2019.

[32]  Terry Lyons,et al.  A Generalised Signature Method for Multivariate Time Series Feature Extraction , 2021 .

[33]  Terry Lyons,et al.  Learning from the past, predicting the statistics for the future, learning an evolving system , 2013, 1309.0260.

[34]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[35]  Lorenzo Rosasco,et al.  Generalization Properties of Learning with Random Features , 2016, NIPS.

[36]  Martin Hairer,et al.  A Course on Rough Paths , 2020, Universitext.

[37]  Marcin Korytkowski,et al.  Convolutional Neural Networks for Time Series Classification , 2017, ICAISC.