Stacked Denoising Auto-Encoders for Short-Term Time Series Forecasting

In this chapter, a study of deep learning of time-series forecasting techniques is presented. Using Stacked Denoising Auto-Encoders, it is possible to disentangle complex characteristics in time series data. The effects of complete and partial fine-tuning are shown. SDAE prove to be able to train deeper models, and consequently to learn more complex characteristics in the data. Hence, these models are able to generalize better. Pre-trained models show a better generalization when used without covariates. The learned weights show to be sparse, suggesting future exploration and research lines.

[1]  Carlos Martín-Vide,et al.  Statistical Language and Speech Processing , 2014, Lecture Notes in Computer Science.

[2]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[3]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[4]  Christopher D. Manning,et al.  Bilingual Word Embeddings for Phrase-Based Machine Translation , 2013, EMNLP.

[5]  Yoshua Bengio,et al.  Deep Learning of Representations: Looking Forward , 2013, SLSP.

[6]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[7]  Jing Gao,et al.  Multistep-Ahead Time Series Prediction , 2006, PAKDD.

[8]  Sergio Vega Sánchez,et al.  Science behind and beyond the solar decathlon Europe 2012 competition , 2014 .

[9]  Pascal Vincent,et al.  Higher Order Contractive Auto-Encoder , 2011, ECML/PKDD.

[10]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[11]  António E. Ruano,et al.  Neural networks based predictive control for thermal comfort and energy savings in public buildings , 2012 .

[12]  Richard A. Davis,et al.  Introduction to time series and forecasting , 1998 .

[13]  Yoshua Bengio,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2010, AISTATS.

[14]  Peter A. Flach,et al.  Subgroup Discovery with CN2-SD , 2004, J. Mach. Learn. Res..

[15]  Juan Pardo,et al.  Time-Series Forecasting of Indoor Temperature Using Pre-trained Deep Neural Networks , 2013, ICANN.

[16]  Günther Palm,et al.  Artificial Neural Networks and Machine Learning – ICANN 2013 , 2013, Lecture Notes in Computer Science.

[17]  Manuel Alcázar-Ortega,et al.  New artificial neural network prediction method for electrical consumption forecasting based on buil , 2011 .

[18]  James W. Taylor.  Exponential smoothing with a damped multiplicative trend , 2003 .

[19]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[20]  Honglak Lee,et al.  Sparse deep belief net model for visual area V2 , 2007, NIPS.

[21]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[22]  Juan Pardo,et al.  Towards Energy Efficiency: Forecasting Indoor Temperature via Multivariate Analysis , 2013, ArXiv.

[23]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[24]  Yoshua Bengio,et al.  Maxout Networks , 2013, ICML.

[25]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[26]  Kunikazu Kobayashi,et al.  Time Series Forecasting Using Restricted Boltzmann Machine , 2012, ICIC.

[27]  Everette S. Gardner,et al.  Exponential smoothing: The state of the art , 1985 .

[28]  Paul E. Utgoff,et al.  Many-Layered Learning , 2002, Neural Computation.

[29]  Yoshua Bengio,et al.  An empirical evaluation of deep architectures on problems with many factors of variation , 2007, ICML '07.

[30]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[31]  Amir F. Atiya,et al.  A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition , 2011, Expert Syst. Appl..

[32]  Pascal Vincent,et al.  The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training , 2009, AISTATS.

[33]  Shen Furao,et al.  Forecasting exchange rate with deep belief networks , 2011, The 2011 International Joint Conference on Neural Networks.

[34]  Pascal Vincent,et al.  Contractive Auto-Encoders: Explicit Invariance During Feature Extraction , 2011, ICML.

[35]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[36]  Michael Y. Hu,et al.  Forecasting with artificial neural networks: The state of the art , 1997 .