Nonlinear time series forecasting with Bayesian neural networks

The Bayesian learning provides a natural way to model the nonlinear structure as the artificial neural networks due to their capability to cope with the model complexity. In this paper, an evolutionary Monte Carlo (MC) algorithm is proposed to train the Bayesian neural networks (BNNs) for the time series forecasting. This approach called as Genetic MC is based on Gaussian approximation with recursive hyperparameter. Genetic MC integrates MC simulations with the genetic algorithms and the fuzzy membership functions. In the implementations, Genetic MC is compared with the traditional neural networks and time series techniques in terms of their forecasting performances over the weekly sales of a Finance Magazine.

[1]  D. Mackay,et al.  HYPERPARAMETERS: OPTIMIZE, OR INTEGRATE OUT? , 1996 .

[2]  C. C. Homes,et al.  Bayesian Radial Basis Functions of Variable Dimension , 1998, Neural Computation.

[3]  Ilya Sutskever,et al.  Learning Recurrent Neural Networks with Hessian-Free Optimization , 2011, ICML.

[4]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[5]  Jouko Lampinen,et al.  Bayesian approach for neural networks--review and case studies , 2001, Neural Networks.

[6]  Peter Müller,et al.  Feedforward Neural Networks for Nonparametric Regression , 1998 .

[7]  Andrew B. Whitford.  Bayesian Methods: A Social and Behavioral Sciences Approach , 2003, Journal of Politics.

[8]  Wray L. Buntine,et al.  Bayesian Back-Propagation , 1991, Complex Syst..

[9]  Peter M. Williams,et al.  Bayesian Regularization and Pruning Using a Laplace Prior , 1995, Neural Computation.

[10]  Chris Chatfield,et al.  Time series forecasting with neural networks: a comparative study using the air line data , 2008 .

[11]  J. MacKinnon,et al.  Estimation and inference in econometrics , 1994 .

[12]  Alan D. Marrs.  An Application of Reversible-Jump MCMC to Multivariate Spherical Gaussian Mixtures , 1997, NIPS.

[13]  Nikolay I. Nikolaev,et al.  Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling , 2010, IEEE Transactions on Neural Networks.

[14]  Jeff Gill,et al.  Bayesian Methods : A Social and Behavioral Sciences Approach , 2002 .

[15]  Yuanchang Xie,et al.  Predicting motor vehicle collisions using Bayesian neural network models: an empirical analysis. , 2007, Accident; analysis and prevention.

[16]  James W. Taylor,et al.  An evaluation of Bayesian techniques for controlling model complexity and selecting inputs in a neural network for short-term load forecasting , 2010, Neural Networks.

[17]  Amine Bermak,et al.  Gaussian process for nonstationary time series prediction , 2004, Comput. Stat. Data Anal..

[18]  Giovanna Castellano,et al.  An iterative pruning algorithm for feedforward neural networks , 1997, IEEE Trans. Neural Networks.

[19]  Zbigniew Michalewicz,et al.  Genetic algorithms + data structures = evolution programs (2nd, extended ed.) , 1994 .

[20]  Gomes de Freitas,et al.  Bayesian methods for neural networks , 2000 .

[21]  Bruce Hajek,et al.  Optimization by simulated annealing: a time-complexity analysis , 1987 .

[22]  J. Vanhatalo,et al.  MCMC Methods for MLP-network and Gaussian Process and Stuff – A documentation for Matlab Toolbox , 2006 .

[23]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[24]  David Mackay,et al.  Probable networks and plausible predictions - a review of practical Bayesian methods for supervised neural networks , 1995 .

[25]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[26]  A. T. C. Goh,et al.  Nonlinear modeling with confidence estimation using Bayesian neural networks , 2004 .

[27]  W. Fuller,et al.  Introduction to Statistical Time Series (2nd ed.) , 1997 .

[28]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[29]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[30]  Tshilidzi Marwala.  Bayesian training of neural networks using genetic programming , 2007, Pattern Recognit. Lett..

[31]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[32]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[33]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[34]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[35]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[36]  T. Ross.  Fuzzy Logic with Engineering Applications , 1994 .

[37]  Faming Liang,et al.  Bayesian neural networks for nonlinear time series forecasting , 2005, Stat. Comput..

[38]  W. Wong,et al.  Real-Parameter Evolutionary Monte Carlo With Applications to Bayesian Mixture Models , 2001 .

[39]  M. Medeiros,et al.  Linear models, smooth transition autoregressions, and neural networks for forecasting macroeconomic time series: A re-examination , 2005 .

[40]  Radford M. Neal.  Bayesian training of backpropagation networks by the hybrid Monte-Carlo method , 1992 .

[41]  Chris Chatfield,et al.  Introduction to Statistical Time Series. , 1976 .

[42]  Federico Girosi,et al.  On the Relationship between Generalization Error, Hypothesis Complexity, and Sample Complexity for Radial Basis Functions , 1996, Neural Computation.

[43]  Michael S. Goodrich.  Markov Chain Monte Carlo Bayesian Learning for Neural Networks , 2011 .

[44]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[45]  Richard M. Golden,et al.  Mathematical Methods for Neural Network Analysis and Design , 1996 .

[46]  Martin T. Hagan,et al.  Neural network design , 1995 .

[47]  Narasimhan Sundararajan,et al.  A generalized growing and pruning RBF (GGAP-RBF) neural network for function approximation , 2005, IEEE Transactions on Neural Networks.

[48]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[49]  Anthony T. C. Goh,et al.  A hybrid Bayesian back‐propagation neural network approach to multivariate modelling , 2003 .