Noisy Time Series Prediction using Symbolic Representation and Recurrent Neural Network Grammatical Inference

Financial forecasting is an example of a signal processing problem which is challenging due to small sample sizes, high noise, non-stationarity, and no n-linearity. Neural networks have been very successful in a number of signal processing applications. We discuss fundamental limita- tions and inherent difficulties when using neural networks f or the processing of high noise, small sample size signals. We introduce a new intelligent signal processing method which addresses the difficulties. The method uses conversion into a symbolic representation with a self-organizing map, and grammatical inference with recurrent neural networks. We apply the method to the pre- diction of daily foreign exchange rates, addressing difficu lties with non-stationarity, overfitting, and unequal a priori class probabilities, and we find significant predictability in comprehensive experiments covering 5 different foreign exchange rates. The method correctly predicts the direc- tion of change for the next day with an error rate of 47.1%. The error rate reduces to around 40% when rejecting examples where the system has low confidence i n its prediction. The symbolic representation aids the extraction of symbolic knowledge from the recurrent neural networks in the form of deterministic finite state automata. These autom ata explain the operation of the sys- tem and are often relatively simple. Rules related to well known behavior such as trend following and mean reversal are extracted.

[1]  Masumi Ishikawa,et al.  Rule extraction by successive regularization , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[2]  Yaser S. Abu-Mostafa,et al.  Learning from hints in neural networks , 1990, J. Complex..

[3]  Huan Liu,et al.  Symbolic Representation of Neural Networks , 1996, Computer.

[4]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[5]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[6]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[7]  C. Lee Giles,et al.  Higher Order Recurrent Networks and Grammatical Inference , 1989, NIPS.

[8]  C. Lee Giles,et al.  Extraction of rules from discrete-time recurrent neural networks , 1996, Neural Networks.

[9]  Vra Krkov.  Kolmogorov's Theorem Is Relevant , 1991, Neural Computation.

[10]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[11]  C. Lee Giles,et al.  Extraction, Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks , 1993 .

[12]  Raymond L. Watrous,et al.  Induction of Finite-State Languages Using Second-Order Recurrent Networks , 1992, Neural Computation.

[13]  Huan Liu,et al.  Symbolic presentation of , 1996 .

[14]  D. Rumelhart,et al.  Predicting sunspots and exchange rates with connectionist networks , 1991 .

[15]  R. Bellman,et al.  V. Adaptive Control Processes , 1964 .

[16]  James L. McClelland,et al.  Finite State Automata and Simple Recurrent Networks , 1989, Neural Computation.

[17]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[18]  Sandiway Fong,et al.  Can recurrent neural networks learn natural language grammars? , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[19]  Padhraic Smyth,et al.  Discrete recurrent neural networks for grammatical inference , 1994, IEEE Trans. Neural Networks.

[20]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[21]  F. Eugene.  FAMA, . The Behavior of Stock-Market Prices, Journal of Business, , . , 1965 .

[22]  Andreas S. Weigend,et al.  Time Series Prediction: Forecasting the Future and Understanding the Past , 1994 .

[23]  G. Yule.  On a Method of Investigating Periodicities in Disturbed Series, with Special Reference to Wolfer's Sunspot Numbers , 1927 .

[24]  András Faragó,et al.  Strong universal consistency of neural network classifiers , 1993, IEEE Trans. Inf. Theory.

[25]  Bruce W. Suter,et al.  The multilayer perceptron as an approximation to a Bayes optimal discriminant function , 1990, IEEE Trans. Neural Networks.

[26]  Yoichi Hayashi,et al.  A Neural Expert System with Automated Extraction of Fuzzy If-Then Rules , 1990, NIPS.

[27]  E. Fama.  The Behavior of Stock-Market Prices , 1965 .

[28]  C. Lee Giles,et al.  Extracting and Learning an Unknown Grammar with Recurrent Neural Networks , 1991, NIPS.

[29]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[30]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[31]  Yoshua Bengio,et al.  Neural networks for speech and sequence recognition , 1996 .

[32]  Jerome H. Friedman,et al.  An Overview of Predictive Learning and Function Approximation , 1994 .

[33]  C. Granger,et al.  Forecasting Economic Time Series. , 1988 .

[34]  John E. Moody,et al.  Note on Learning Rate Schedules for Stochastic Optimization , 1990, NIPS.

[35]  J. Elman.  Distributed Representations, Simple Recurrent Networks, And Grammatical Structure , 1991 .

[36]  Lester Ingber,et al.  Statistical Mechanics of Nonlinear Nonequilibrium Financial Markets: Applications to Optimized Trading , 1996 .

[37]  Lawrence D. Jackel,et al.  Large Automatic Learning, Rule Extraction, and Generalization , 1987, Complex Syst..

[38]  Toshio Odanaka,et al.  ADAPTIVE CONTROL PROCESSES , 1990 .

[39]  J. Moody.  Economic forecasting : challenges and neural network solutions , 1995 .

[40]  Stephen L Taylor,et al.  Modelling Financial Time Series , 1987 .

[41]  Jude Shavlik,et al.  THE EXTRACTION OF REFINED RULES FROM KNOWLEDGE BASED NEURAL NETWORKS , 1993 .

[42]  Michael C. Mozer,et al.  Rule Induction through Integrated Symbolic and Subsymbolic Processing , 1991, NIPS.

[43]  E. Fama.  EFFICIENT CAPITAL MARKETS: A REVIEW OF THEORY AND EMPIRICAL WORK* , 1970 .

[44]  Věra Kůrková.  Kolmogorov's theorem , 1998 .

[45]  Michael C. Mozer,et al.  Template-Based Algorithms for Connectionist Rule Extraction , 1994, NIPS.

[46]  Burton G. Malkiel,et al.  Efficient Market Hypothesis , 1991 .

[47]  守屋 悦朗,et al.  J.E.Hopcroft, J.D. Ullman 著, "Introduction to Automata Theory, Languages, and Computation", Addison-Wesley, A5変形版, X+418, \6,670, 1979 , 1980 .