Continuous history compression

Neural networks have proven poor at learning the structure in complex and extended temporal sequences in which contingencies among elements can span long time lags. The principle of history compression 18] provides a means of transforming long sequences with redundant information into equivalent shorter sequences; the shorter sequences are more easily manipulated and learned by neural networks. The principle states that expected sequence elements can be removed from the sequence to form an equivalent, more compact sequence without loss of information. The principle was embodied in a neural net predictive architecture that attempted to anticipate the next element of a sequence given the previous elements. If the prediction was accurate, the next element was discarded; otherwise, it was passed on to a second network that processed the sequence in some fashion (e.g., recognition, classiication, autoencoding, etc.). As originally proposed, a binary judgement was made as to the predictability of each element. Here, we describe a contininuous version of history compression in which elements are discarded in a graded fashion dependent on their predictability, embodied by their (Shannon) information. We implement continuous history compression using a RAAM architecture, yielding a class of sequence learning algorithms that are both entirely local and still able to bridge long time lags between correlated events.