Online Symbolic-Sequence Prediction with Recurrent Neural Networks

This paper studies the use of recurrent neural networks for predicting the next symbol in a sequence. The focus is on online prediction, a task much harder than the classical offline grammatical inference with neural networks. Different kinds of sequence sources are considered: finitestate machines, chaotic sources, and texts in human language. Two algorithms are used for network training: real-time recurrent learning and the decoupled extended Kalman filter. + Abbreviations CR: Compression ratio. DEKF: Decoupled extended Kalman filter. RNN: Recurrent neural network. RTRL: Real-time recurrent learning. SRN: Simple recurrent network. Objective Our objective is to evaluate the performance of discrete-time recurrent neural networks (RNN) in online prediction. RNN are trained to predict in real-time the next symbol in a sequence. The network output may then be considered as an estimation of next-symbol probabilities. Arithmetic compression is used to measure the quality of the predictor.