Multi-Time Resolution Ensemble LSTMs for Enhanced Feature Extraction in High-Rate Time Series

Systems experiencing high-rate dynamic events, termed high-rate systems, typically undergo accelerations of amplitudes higher than 100 g-force in less than 10 ms. Examples include adaptive airbag deployment systems, hypersonic vehicles, and active blast mitigation systems. Given their critical functions, accurate and fast modeling tools are necessary for ensuring the target performance. However, the unique characteristics of these systems, which consist of (1) large uncertainties in the external loads, (2) high levels of non-stationarities and heavy disturbances, and (3) unmodeled dynamics generated from changes in system configurations, in combination with the fast-changing environments, limit the applicability of physical modeling tools. In this paper, a deep learning algorithm is used to model high-rate systems and predict their response measurements. It consists of an ensemble of short-sequence long short-term memory (LSTM) cells which are concurrently trained. To empower multi-step ahead predictions, a multi-rate sampler is designed to individually select the input space of each LSTM cell based on local dynamics extracted using the embedding theorem. The proposed algorithm is validated on experimental data obtained from a high-rate system. Results showed that the use of the multi-rate sampler yields better feature extraction from non-stationary time series compared with a more heuristic method, resulting in significant improvement in step ahead prediction accuracy and horizon. The lean and efficient architecture of the algorithm results in an average computing time of 25 μs, which is below the maximum prediction horizon, therefore demonstrating the algorithm’s promise in real-time high-rate applications.

[1]  Fraser,et al.  Independent coordinates for strange attractors from mutual information. , 1986, Physical review. A, General physics.

[2]  Mohammad R. Jahanshahi,et al.  Autonomous vision-based damage chronology for spatiotemporal condition assessment of civil infrastructure using unmanned aerial vehicle , 2020 .

[3]  Jaromír Vrbka,et al.  Bankruptcy or Success? The Effective Prediction of a Company’s Financial Development Using LSTM , 2020, Sustainability.

[4]  Simon Laflamme,et al.  An Experimental Test Bed for Developing High-Rate Structural Health Monitoring Methods , 2018 .

[5]  Hang Lei,et al.  Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization , 2019 .

[6]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[7]  Yoshua Bengio,et al.  Mutual Information Neural Estimation , 2018, ICML.

[8]  Petros Koumoutsakos,et al.  Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks , 2018, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[9]  David S. Broomhead,et al.  Delay Embeddings for Forced Systems. II. Stochastic Forcing , 2003, J. Nonlinear Sci..

[10]  J. A. Stewart,et al.  Nonlinear Time Series Analysis , 2015 .

[11]  Antoine Cully,et al.  Robots that can adapt like animals , 2014, Nature.

[12]  Sung-Bae Cho,et al.  Evolutionary Optimization of Hyperparameters in Deep Learning Models , 2019, 2019 IEEE Congress on Evolutionary Computation (CEC).

[13]  J. Beyerer,et al.  Optimisation of manufacturing process parameters using deep neural networks as surrogate models , 2018 .

[14]  Nikolay Laptev,et al.  Deep and Confident Prediction for Time Series at Uber , 2017, 2017 IEEE International Conference on Data Mining Workshops (ICDMW).

[15]  Xianfu Chen,et al.  Deep Learning with Long Short-Term Memory for Time Series Prediction , 2018, IEEE Communications Magazine.

[16]  Nicolas Thome,et al.  Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models , 2019, NeurIPS.

[17]  Homayoun Najjaran,et al.  Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review , 2020, Sensors.

[18]  José Cristóbal Riquelme Santos,et al.  Coronavirus Optimization Algorithm: A bioinspired metaheuristic based on the COVID-19 propagation model , 2020, Big Data.

[19]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[20]  Lana X. Garmire,et al.  DeepImpute: an accurate, fast, and scalable deep neural network method to impute single-cell RNA-seq data , 2018, Genome Biology.

[21]  Meinard Müller,et al.  Information retrieval for music and motion , 2007 .

[22]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986 .

[24]  Ling Yang,et al.  DSTP-RNN: a dual-stage two-phase attention-based recurrent neural networks for long-term and multivariate time series prediction , 2019, Expert Syst. Appl..

[25]  Bryan Lim,et al.  Time-series forecasting with deep learning: a survey , 2020, Philosophical Transactions of the Royal Society A.

[26]  Richard Bowden,et al.  A Survey of Deep Learning Applications to Autonomous Vehicle Control , 2019, IEEE Transactions on Intelligent Transportation Systems.

[27]  Anuj Karpatne,et al.  Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling , 2017, ArXiv.

[28]  José Cristóbal Riquelme Santos,et al.  An Experimental Review on Deep Learning Architectures for Time Series Forecasting , 2020, Int. J. Neural Syst..

[29]  Igor Melnyk,et al.  Deep learning algorithm for data-driven simulation of noisy dynamical system , 2018, J. Comput. Phys..

[30]  Marco Gori,et al.  Integrating Prior Knowledge into Deep Learning , 2017, 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA).

[31]  Simon Laflamme,et al.  Variable input observer for nonstationary high-rate dynamic systems , 2018, Neural Computing and Applications.

[32]  Simon Laflamme,et al.  Introduction to State Estimation of High-Rate System Dynamics , 2018, Sensors.

[33]  Guokun Lai,et al.  Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks , 2017, SIGIR.

[34]  Simon Laflamme,et al.  Online parameter estimation under non-persistent excitations for high-rate dynamic systems , 2021 .

[35]  Alan Bundy,et al.  Dynamic Time Warping , 1984 .

[36]  J. Stark,et al.  Delay Embeddings for Forced Systems. I. Deterministic Forcing , 1999 .

[37]  Hsiang-Fu Yu,et al.  Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting , 2019, NeurIPS.

[38]  J. Coykendall,et al.  An embedding theorem , 2011 .

[39]  Kimiaki Shirahama,et al.  Deep Transfer Learning for Time Series Data Based on Sensor Modality Classification , 2020, Sensors.

[40]  Jacob Dodson,et al.  Millisecond model updating for structures experiencing unmodeled high-rate dynamic events , 2020 .

[41]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[42]  Jacob Dodson,et al.  Study of input space for state estimation of high‐rate dynamics , 2018 .

[43]  W. Gilpin.  Deep reconstruction of strange attractors from time series , 2020, NeurIPS.

[44]  Xiao Liang,et al.  Uncertainty‐assisted deep vision structural health monitoring , 2020, Comput. Aided Civ. Infrastructure Eng..

[45]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[46]  Matthias Seeger,et al.  Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning , 2019, NeurIPS.

[47]  Alex Graves,et al.  Supervised Sequence Labelling , 2012 .

[48]  Yinhai Wang,et al.  Stacked Bidirectional and Unidirectional LSTM Recurrent Neural Network for Forecasting Network-wide Traffic State with Missing Values , 2020, Transportation Research Part C: Emerging Technologies.

[49]  Divish Rengasamy,et al.  Deep Learning with Dynamically Weighted Loss Function for Sensor-Based Prognostics and Health Management , 2020, Sensors.