Pay Attention to Evolution: Time Series Forecasting with Deep Graph-Evolution Learning

Time-series forecasting is one of the most active research topics in artificial intelligence. A still open gap in that literature is that statistical and ensemble learning approaches systematically present lower predictive performance than deep learning methods. They generally disregard the data sequence aspect entangled with multivariate data represented in more than one time series. Conversely, this work presents a novel neural network architecture for time-series forecasting that combines the power of graph evolution with deep recurrent learning on distinct data distributions; we named our method Recurrent Graph Evolution Neural Network (ReGENN). The idea is to infer multiple multivariate relationships between co-occurring time-series by assuming that the temporal data depends not only on inner variables and intra-temporal relationships (i.e., observations from itself) but also on outer variables and inter-temporal relationships (i.e., observations from other-selves). An extensive set of experiments was conducted comparing ReGENN with dozens of ensemble methods and classical statistical ones, showing sound improvement of up to 64.87% over the competing algorithms. Furthermore, we present an analysis of the intermediate weights arising from ReGENN, showing that by looking at inter and intra-temporal relationships simultaneously, time-series forecasting is majorly improved if paying attention to how multiple multivariate data synchronously evolve.

[1]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[2]  Alexander J. Smola,et al.  Support Vector Method for Function Approximation, Regression Estimation and Signal Processing , 1996, NIPS.

[3]  Anil K. Jain,et al.  Artificial Neural Networks: A Tutorial , 1996, Computer.

[4]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[5]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[6]  Neil Davey,et al.  Input window size and neural network predictors , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[7]  Neil Davey,et al.  Time Series Prediction and Neural Networks , 2001, J. Intell. Robotic Syst..

[8]  Michael Y. Hu,et al.  A simulation study of artificial neural networks for nonlinear time-series forecasting , 2001, Comput. Oper. Res..

[9]  Eamonn J. Keogh,et al.  Segmenting Time Series: A Survey and Novel Approach , 2002 .

[10]  David Veredas,et al.  Temporal Aggregation of Univariate and Multivariate Time Series Models: A Survey , 2008 .

[11]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[12]  Charlene Elliott,et al.  Taste™ , 2012 .

[13]  G. Moody,et al.  Predicting in-hospital mortality of ICU patients: The PhysioNet/Computing in cardiology challenge 2012 , 2012, 2012 Computing in Cardiology.

[14]  Marc F. P. Bierkens,et al.  Rising river flows throughout the twenty-first century in two Himalayan glacierized watersheds , 2013 .

[15]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[16]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[17]  Saeed Amizadeh,et al.  Generic and Scalable Framework for Automated Time-series Anomaly Detection , 2015, KDD.

[18]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[19]  Jürgen Schmidhuber,et al.  Highway Networks , 2015, ArXiv.

[20]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[21]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[23]  Jimeng Sun,et al.  Multi-layer Representation Learning for Medical Concepts , 2016, KDD.

[24]  Witold Pedrycz,et al.  Multivariate time series anomaly detection: A framework of Hidden Markov Models , 2017, Appl. Soft Comput..

[25]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[26]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[27]  Garrison W. Cottrell,et al.  A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction , 2017, IJCAI.

[28]  Jürgen Schmidhuber,et al.  Recurrent Highway Networks , 2016, ICML.

[29]  Fan Zhang,et al.  A review on time series forecasting techniques for building energy consumption , 2017 .

[30]  B. Hunt,et al.  Estimation of the maximum annual number of North Atlantic tropical cyclones using climate models , 2018, Science Advances.

[31]  Zhanxing Zhu,et al.  Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting , 2017, IJCAI.

[32]  Guokun Lai,et al.  Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks , 2017, SIGIR.

[33]  Xavier Bresson,et al.  Structured Sequence Modeling with Graph Convolutional Recurrent Networks , 2016, ICONIP.

[34]  Cyrus Shahabi,et al.  Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting , 2017, ICLR.

[35]  Steven C. H. Hoi,et al.  Online Deep Learning: Learning Deep Neural Networks on the Fly , 2017, IJCAI.

[36]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[37]  Hao Ma,et al.  GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs , 2018, UAI.

[38]  Anna Veronika Dorogush,et al.  CatBoost: unbiased boosting with categorical features , 2017, NeurIPS.

[39]  Anna Veronika Dorogush,et al.  CatBoost: gradient boosting with categorical features support , 2018, ArXiv.

[40]  José F. Rodrigues,et al.  Patient trajectory prediction in the Mimic-III dataset, challenges and pitfalls , 2019, ArXiv.

[41]  Ao Tang,et al.  DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting , 2019, CIKM.

[42]  Richard W. Vuduc,et al.  Temporal phenotyping of medically complex children via PARAFAC2 tensor factorization , 2019, J. Biomed. Informatics.

[43]  T. Velavan,et al.  The COVID‐19 epidemic , 2020, Tropical medicine & international health : TM & IH.

[44]  Eric Ghysels,et al.  Machine Learning Time Series Regressions With an Application to Nowcasting , 2019, Journal of Business & Economic Statistics.

[45]  Eric Ghysels,et al.  Machine Learning Panel Data Regressions with an Application to Nowcasting Price Earnings Ratios , 2020, SSRN Electronic Journal.

[46]  Erol Egrioglu,et al.  Picture fuzzy time series: Defining, modeling and creating a new forecasting method , 2020, Eng. Appl. Artif. Intell..

[47]  R. Barthelmie,et al.  Climate change impacts on wind power generation , 2020, Nature Reviews Earth & Environment.

[48]  J. Sillmann,et al.  Better seasonal forecasts for the renewable energy industry , 2020 .

[49]  D. Roy,et al.  Spatially and temporally complete Landsat reflectance time series modelling: The fill-and-fit approach , 2020 .

[50]  Zhiqiang Yang,et al.  Continuous monitoring of land disturbance based on Landsat time series , 2020, Remote Sensing of Environment.

[51]  Marcello Ienca,et al.  On the responsible use of digital data to tackle the COVID-19 pandemic , 2020, Nature Medicine.

[52]  D. Roy,et al.  Robust Landsat-based crop time series modelling , 2020 .

[53]  Tianrui Li,et al.  Multivariate time series forecasting via attention-based encoder-decoder framework , 2020, Neurocomputing.

[54]  Yiu Chung Lau,et al.  Temporal dynamics in viral shedding and transmissibility of COVID-19 , 2020, Nature Medicine.

[55]  Sihem Amer-Yahia,et al.  Lig-Doctor: Real-World Clinical Prognosis using a Bi-Directional Neural Network , 2020, 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS).

[56]  Prateek Pandey,et al.  Stock Market Prediction Using Optimized Deep-ConvLSTM Model , 2020, Big Data.

[57]  John T. Hancock,et al.  Performance of CatBoost and XGBoost in Medicare Fraud Detection , 2020, 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA).

[58]  Ling Yang,et al.  DSTP-RNN: a dual-stage two-phase attention-based recurrent neural networks for long-term and multivariate time series prediction , 2019, Expert Syst. Appl..

[59]  Jean-Louis Mugnier,et al.  The impact of climate change and glacier mass loss on the hydrology in the Mont-Blanc massif , 2020, Scientific Reports.

[60]  Yu Liu,et al.  T-GCN: A Temporal Graph Convolutional Network for Traffic Prediction , 2018, IEEE Transactions on Intelligent Transportation Systems.

[61]  Kaizhu Huang,et al.  Towards Better Forecasting by Fusing Near and Distant Future Visions , 2019, AAAI.

[62]  Ian G. Cowx,et al.  Effects of climate and land-use changes on fish catches across lakes at a global scale , 2020, Nature Communications.

[63]  L. Mombaerts,et al.  An interpretable mortality prediction model for COVID-19 patients , 2020, Nature Machine Intelligence.

[64]  E. Dong,et al.  An interactive web-based dashboard to track COVID-19 in real time , 2020, The Lancet Infectious Diseases.

[65]  Jimeng Sun,et al.  TASTE: temporal and static tensor factorization for phenotyping electronic health records , 2019, CHIL.

[66]  Zhaoxia Yu,et al.  Change-point detection using spectral PCA for multivariate time series , 2021 .

[67]  Hui Xiong,et al.  Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting , 2020, AAAI.

[68]  K. Vijayakumar,et al.  Stock market analysis using candlestick regression and market trend prediction (CKRM) , 2020, Journal of Ambient Intelligence and Humanized Computing.

[69]  Alexander M. Niziolek,et al.  A framework to predict the price of energy for the end-users with applications to monetary and energy policies , 2021, Nature communications.

[70]  J. Aiello The Symbolic Instrumentalisation of the Face Mask in (De)legitimising Discourse: A Critical Study of User-Generated Online Content at the Onset of the Covid-19 Pandemic in the US , 2021 .

[71]  Sihem Amer-Yahia,et al.  LIG-Doctor: Efficient patient trajectory prediction using bidirectional minimal gated-recurrent networks , 2021, Inf. Sci..