On-line stochastic processes in data compression

Program Authorized to OOer Degree Date In presenting this dissertation in partial fullllment of the requirements for the Doctoral degree at the University of Washington, I agree that the Library shall make its copies freely available for inspection. I further agree that extensive copying of this dissertation is allowable only for scholarly purposes, consistent with \fair use" as prescribed in the U.S. to whom the author has granted \the right to reproduce and sell (a) copies of the manuscript in microform and/or (b) printed copies of the manuscript made from microform." The ability to predict the future based upon the past in nite-alphabet sequences has many applications, including communications, data security, pattern recognition, and natural language processing. By Shannon's theory and the breakthrough development of arithmetic coding, any sequence, a 1 a 2 a n , can be encoded in a number of bits that is essentially equal to the minimal information-lossless codelength, P i ? log 2 p(a i ja 1 a i?1). The goal of universal on-line modeling, and therefore of universal data compression, is to deduce the model of the input sequence a 1 a 2 a n that can estimate each p(a i ja 1 a i?1) knowing only a 1 a 2 a i?1 so that the expected value of ? log p(a i ja 1 a i?1) is minimized. Thus, data compression has become both a routine application of on-line modeling techniques and a means for accurately measuring their empirical performance. The on-line modeling algorithm, Prediction By Partial Matching (PPM), has set the performance standard in data compression research since its introduction in 1984. PPM's success stems from its ad hoc probability estimator, which dynamically blends distinct frequency distributions contained in a single model into a probability estimate for each input symbol. Meanwhile, the most conclusive asymptotic results use an information-theoretic metric to dynamically select a model from a set of competing models, and then use that selected model to estimate the currently scanned symbol's probability. Our hypothesis is that these apparently unrelated approaches can be combined to produce a semantically coherent technique that is arguably universal and which consistently outperforms existing techniques on actual data. To prove our hypothesis, we rst give a semantics that uniies both forms of on-line modeling. Then we show how related but linguistically distinct model families t the semantics, and give a new frequency update mechanism that is …