Modeling time series data with deep Fourier neural networks

Abstract We present a method for training a deep neural network containing sinusoidal activation functions to fit to time-series data. Weights are initialized using a fast Fourier transform, then trained with regularization to improve generalization. A simple dynamic parameter tuning method is employed to adjust both the learning rate and the regularization term, such that both stability and efficient training are achieved. We show how deeper layers can be utilized to model the observed sequence using a sparser set of sinusoid units, and how non-uniform regularization can improve generalization by promoting the shifting of weight toward simpler units. The method is demonstrated with time-series problems to show that it leads to effective extrapolation of nonlinear trends.

[1]  Ravi Sankar,et al.  Time Series Prediction Using Support Vector Machines: A Survey , 2009, IEEE Computational Intelligence Magazine.

[2]  Francesco Mondada,et al.  Automatic creation of an autonomous agent: genetic evolution of a neural-network driven robot , 1994 .

[3]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[4]  Armando Blanco,et al.  A real-coded genetic algorithm for training recurrent neural networks , 2001, Neural Networks.

[5]  Liangyue Cao,et al.  Predicting chaotic time series with wavelet networks , 1995 .

[6]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[7]  Milton S. Boyd,et al.  Designing a neural network for forecasting financial and economic time series , 1996, Neurocomputing.

[8]  Adrian Silvescu,et al.  Fourier neural networks , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[9]  Neil Davey,et al.  Time Series Prediction and Neural Networks , 2001, J. Intell. Robotic Syst..

[10]  Michael S. Gashler,et al.  Waffles: A Machine Learning Toolkit , 2011, J. Mach. Learn. Res..

[11]  Guoqiang Peter Zhang,et al.  Time series forecasting using a hybrid ARIMA and neural network model , 2003, Neurocomputing.

[12]  Richard S. Sutton,et al.  Neural networks for control , 1990 .

[13]  Amir B. Geva,et al.  ScaleNet-multiscale neural-network architecture for time series prediction , 1998, IEEE Trans. Neural Networks.

[14]  Lilong Cai,et al.  Fourier-Neural-Network-Based Learning Control for a Class of Nonlinear Systems With Flexible Components , 2009, IEEE Transactions on Neural Networks.

[15]  Abdulhamit Subasi,et al.  Wavelet neural network classification of EEG signals by using AR model with MLE preprocessing , 2005, Neural Networks.

[16]  Tony R. Martinez,et al.  The general inefficiency of batch training for gradient descent learning , 2003, Neural Networks.

[17]  Jürgen Schmidhuber,et al.  A committee of neural networks for traffic sign classification , 2011, The 2011 International Joint Conference on Neural Networks.

[18]  Jiwen Dong,et al.  Time-series prediction using a local linear wavelet neural network , 2006, Neurocomputing.

[19]  Rob J Hyndman,et al.  25 years of time series forecasting , 2006 .

[20]  Eduardo D. Sontag,et al.  Neural Networks for Control , 1993 .

[21]  H. S. Tan.  Fourier neural networks and generalized single hidden layer networks in aircraft engine fault diagnostics , 2006 .

[22]  L. Tsimring,et al.  The analysis of observed chaotic data in physical systems , 1993 .

[23]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[24]  Fionn Murtagh,et al.  Combining Neural Network Forecasts on Wavelet-transformed Time Series , 1997, Connect. Sci..

[25]  Qinghua Zhang,et al.  Wavelet networks , 1992, IEEE Trans. Neural Networks.

[26]  Chuntian Cheng,et al.  Using support vector machines for long-term discharge prediction , 2006 .

[27]  Michael Y. Hu,et al.  Forecasting with artificial neural networks: The state of the art , 1997 .

[28]  Michael C. Mozer,et al.  A Focused Backpropagation Algorithm for Temporal Pattern Recognition , 1989, Complex Syst..

[29]  Shouyang Wang,et al.  Forecasting stock market movement direction with support vector machine , 2005, Comput. Oper. Res..

[30]  Pierre Roussel-Ragot,et al.  Training recurrent neural networks: why and how? An illustration in dynamical process modeling , 1994, IEEE Trans. Neural Networks.

[31]  Manuel P. Cuéllar,et al.  An Application of Non-Linear Programming to Train Recurrent Neural Networks in Time Series Prediction Problems , 2005, ICEIS.

[32]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[33]  Garrison W. Cottrell,et al.  New Life for Neural Networks , 2006, Science.

[34]  Marc'Aurelio Ranzato,et al.  Building high-level features using large scale unsupervised learning , 2011, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[35]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[37]  L. F. Mingo,et al.  FOURIER NEURAL NETWORKS: AN APPROACH WITH SINUSOIDAL ACTIVATION FUNCTIONS 1 , 2003 .

[38]  Tony R. Martinez,et al.  Temporal nonlinear dimensionality reduction , 2011, The 2011 International Joint Conference on Neural Networks.

[39]  PAUL J. WERBOS,et al.  Generalization of backpropagation with application to a recurrent gas market model , 1988, Neural Networks.

[40]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[41]  Keechul Jung,et al.  GPU implementation of neural networks , 2004, Pattern Recognit..

[42]  Lennart Ljung,et al.  Nonlinear Black Box Modeling in System Identification , 1995 .

[43]  C. J. Franco,et al.  PREDICCIÓN DE SERIES TEMPORALES USANDO MÁQUINAS DE VECTORES DE SOPORTE TIME SERIES PREDICTION USING SUPPORT VECTOR MACHINES , 2010 .