Universal Filtering Via Prediction

We consider the filtering problem, where a finite-alphabet individual sequence is corrupted by a discrete memoryless channel, and the goal is to causally estimate each sequence component based on the past and present noisy observations. We establish a correspondence between the filtering problem and the problem of prediction of individual sequences which leads to the following result: Given an arbitrary finite set of filters, there exists a filter which performs, with high probability, essentially as well as the best in the set, regardless of the underlying noiseless individual sequence. We use this relationship between the problems to derive a filter guaranteed of attaining the "finite-state filterability" of any individual sequence by leveraging results from the prediction problem

[1]  E. Samuel Asymptotic Solutions of the Sequential Compound Decision Problem , 1963 .

[2]  G. Lugosi,et al.  On Prediction of Individual Sequences , 1998 .

[3]  Tsachy Weissman,et al.  Twofold universal prediction schemes for achieving the finite-state predictability of a noisy individual binary sequence , 2001, IEEE Trans. Inf. Theory.

[4]  Neri Merhav,et al.  Hidden Markov processes , 2002, IEEE Trans. Inf. Theory.

[5]  F. R. Gantmakher The Theory of Matrices , 1984 .

[6]  Cun-Hui Zhang,et al.  Compound decision theory and empirical bayes methods , 2003 .

[7]  J. V. Ryzin,et al.  The Sequential Compound Decision Problem with $m \times n$ Finite Loss Matrix , 1966 .

[8]  D. Blackwell An analog of the minimax theorem for vector payoffs. , 1956 .

[9]  Hans S. Witsenhausen,et al.  Indirect rate distortion problems , 1980, IEEE Trans. Inf. Theory.

[10]  H. Robbins,et al.  Asymptotic Solutions of the Compound Decision Problem for Two Completely Specified Distributions , 1955 .

[11]  Robert M. Gray,et al.  A unified approach for encoding clean and noisy sources by means of waveform and autoregressive model vector quantization , 1988, IEEE Trans. Inf. Theory.

[12]  S. Vardeman Admissible solutions of k-extended finite state set and sequence compound decision problems , 1980 .

[13]  Axthonv G. Oettinger,et al.  IEEE Transactions on Information Theory , 1998 .

[14]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[15]  Peter Lancaster,et al.  The theory of matrices , 1969 .

[16]  G. Lugosi,et al.  On Prediction of Individual Sequences , 1998 .

[17]  H. Robbins Asymptotically Subminimax Solutions of Compound Statistical Decision Problems , 1985 .

[18]  James Hannan,et al.  4. APPROXIMATION TO RAYES RISK IN REPEATED PLAY , 1958 .

[19]  Claude E. Shannon,et al.  Channels with Side Information at the Transmitter , 1958, IBM J. Res. Dev..

[20]  David Haussler,et al.  How to use expert advice , 1993, STOC.

[21]  Tsachy Weissman,et al.  Compound Sequential Decisions Against the Well-Informed Antagonist , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Punta del Este.

[22]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[23]  Tsachy Weissman,et al.  Universal discrete denoising: known channel , 2003, IEEE Transactions on Information Theory.

[24]  W. Hoeffding Probability Inequalities for sums of Bounded Random Variables , 1963 .

[25]  Tsachy Weissman,et al.  Universal prediction of individual binary sequences in the presence of noise , 2001, IEEE Trans. Inf. Theory.

[26]  Tsachy Weissman,et al.  On limited-delay lossy coding and filtering of individual sequences , 2002, IEEE Trans. Inf. Theory.

[27]  Vladimir Vovk,et al.  Aggregating strategies , 1990, COLT '90.

[28]  Neri Merhav,et al.  Universal Prediction , 1998, IEEE Trans. Inf. Theory.

[29]  Boris Tsybakov,et al.  Information transmission with additional noise , 1962, IRE Trans. Inf. Theory.

[30]  Abraham Lempel,et al.  Compression of individual sequences via variable-rate coding , 1978, IEEE Trans. Inf. Theory.

[31]  E. Ordentlich,et al.  On-line decision making for a class of loss functions via Lempel-Ziv parsing , 2000, Proceedings DCC 2000. Data Compression Conference.

[32]  Tsachy Weissman,et al.  Discrete universal filtering through incremental parsing , 2004, Data Compression Conference, 2004. Proceedings. DCC 2004.

[33]  Stephen B. Vardeman,et al.  Approximation to Minimum K-Extended Bayes Risk in Sequences of Finite State Decision Problems and Games , 1982 .

[34]  R. Durrett Probability: Theory and Examples , 1993 .

[35]  Neri Merhav,et al.  Universal schemes for sequential decision from individual data sequences , 1993, IEEE Trans. Inf. Theory.

[36]  Thomas M. Cover,et al.  Compound Bayes Predictors for Sequences with Apparent Markov Structure , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[37]  Thomas M. Cover,et al.  Behavior of sequential predictors of binary sequences , 1965 .

[38]  Neri Merhav,et al.  Universal prediction of individual sequences , 1992, IEEE Trans. Inf. Theory.

[39]  Philip Wolfe,et al.  Contributions to the theory of games , 1953 .

[40]  J. V. Ryzin,et al.  The Compound Decision Problem with $m \times n$ Finite Loss Matrix , 1966 .