Universal discrete denoising: known channel

A discrete denoising algorithm estimates the input sequence to a discrete memoryless channel (DMC) based on the observation of the entire output sequence. For the case in which the DMC is known and the quality of the reconstruction is evaluated with a given single-letter fidelity criterion, we propose a discrete denoising algorithm that does not assume knowledge of statistical properties of the input sequence. Yet, the algorithm is universal in the sense of asymptotically performing as well as the optimum denoiser that knows the input sequence distribution, which is only assumed to be stationary. Moreover, the algorithm is universal also in a semi-stochastic setting, in which the input is an individual sequence, and the randomness is due solely to the channel noise. The proposed denoising algorithm is practical, requiring a linear number of register-level operations and sublinear working storage size relative to the input data length.

[1]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[2]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.

[3]  Meir Feder,et al.  A universal finite memory source , 1995, IEEE Trans. Inf. Theory.

[4]  Robert L. Mercer,et al.  Context based spelling correction , 1991, Inf. Process. Manag..

[5]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[6]  Martin Vetterli,et al.  Bridging Compression to Wavelet Thresholding as a Denoising Method , 1997 .

[7]  Balas K. Natarajan.  Filtering random noise via data compression , 1993, [Proceedings] DCC `93: Data Compression Conference.

[8]  Dana Angluin,et al.  Learning Markov chains with variable memory length from noisy output , 1997, COLT '97.

[9]  Konstantinos Konstantinides,et al.  Occam filters for stochastic sources with application to digital images , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[10]  Tsachy Weissman,et al.  Universal prediction of individual binary sequences in the presence of noise , 2001, IEEE Trans. Inf. Theory.

[11]  E. Samuel.  An Empirical Bayes Approach to the Testing of Certain Parametric Hypotheses , 1963 .

[12]  James Hannan,et al.  4. APPROXIMATION TO RAYES RISK IN REPEATED PLAY , 1958 .

[13]  Toby Berger,et al.  Lossy Source Coding , 1998, IEEE Trans. Inf. Theory.

[14]  R. Durrett.  Probability: Theory and Examples , 1993 .

[15]  J. V. Ryzin,et al.  The Compound Decision Problem with $m \times n$ Finite Loss Matrix , 1966 .

[16]  Jacob Ziv,et al.  Distortion-rate theory for individual sequences , 1980, IEEE Trans. Inf. Theory.

[17]  Steven P. Lalley,et al.  Removing the Noise from Chaos Plus Noise , 2001 .

[18]  S. Vardeman.  Admissible Solutions of Finite State Sequence Compound Decision Problems , 1978 .

[19]  E. Samuel.  On Simple Rules for the Compound Decision Problem , 1965 .

[20]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[21]  R. Chang,et al.  On receiver structures for channels having memory , 1966, IEEE Trans. Inf. Theory.

[22]  E. Samuel.  Asymptotic Solutions of the Sequential Compound Decision Problem , 1963 .

[23]  David L. Donoho,et al.  The Kolmogorov Sampler , 2002 .

[24]  Giuseppe Caire,et al.  Almost noiseless joint source channel coding-decoding of sources of memory , 2004 .

[25]  H. Robbins,et al.  Asymptotic Solutions of the Compound Decision Problem for Two Completely Specified Distributions , 1955 .

[26]  Tsachy Weissman,et al.  Discrete universal filtering through incremental parsing , 2004, Data Compression Conference, 2004. Proceedings. DCC 2004.

[27]  Tsachy Weissman,et al.  The minimax distortion redundancy in noisy source coding , 2003, IEEE Trans. Inf. Theory.

[28]  Stephen B. Vardeman,et al.  Approximation to Minimum K-Extended Bayes Risk in Sequences of Finite State Decision Problems and Games , 1982 .

[29]  Tsachy Weissman,et al.  Universal minimax discrete denoising under channel uncertainty , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[30]  Steven P. Lalley,et al.  Beneath the noise, chaos , 1999 .

[31]  Fred J. Damerau,et al.  An examination of undetected typing errors , 1989, Inf. Process. Manag..

[32]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[33]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[34]  Tsachy Weissman,et al.  On limited-delay lossy coding and filtering of individual sequences , 2002, IEEE Trans. Inf. Theory.

[35]  Martin Vetterli,et al.  Adaptive wavelet thresholding for image denoising and compression , 2000, IEEE Trans. Image Process..

[36]  Brian D. O. Anderson,et al.  Asymptotic smoothing errors for hidden Markov models , 2000, IEEE Trans. Signal Process..

[37]  Balas K. Natarajan.  Filtering random noise from deterministic signals via data compression , 1995, IEEE Trans. Signal Process..

[38]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[39]  Neri Merhav,et al.  Universal Prediction , 1998, IEEE Trans. Inf. Theory.

[40]  Michael Gastpar,et al.  To code or not to code , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[41]  Hong-Ye Gao,et al.  Wavelet analysis [for signal processing] , 1996 .

[42]  Tsachy Weissman,et al.  Efficient pruning of bi-directional context trees with applications to universal denoising and compression , 2004, Information Theory Workshop.

[43]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[44]  H. Robbins,et al.  TESTING STATISTICAL HYPOTHESES. THE COMPOUND APPROACH , 1961 .

[45]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[46]  Andrew B. Nobel,et al.  Denoising deterministic time series , 2006, nlin/0604052.

[47]  Peter Lancaster,et al.  The theory of matrices , 1969 .

[48]  Feller William,et al.  An Introduction To Probability Theory And Its Applications , 1950 .

[49]  Tsachy Weissman,et al.  A discrete universal denoiser and its application to binary images , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[50]  Neri Merhav,et al.  Hidden Markov processes , 2002, IEEE Trans. Inf. Theory.

[51]  H. W. Bode,et al.  A Simplified Derivation of Linear Least Square Smoothing and Prediction Theory , 1950, Proceedings of the IRE.

[52]  Neri Merhav,et al.  Universal prediction of individual sequences , 1992, IEEE Trans. Inf. Theory.

[53]  Frans M. J. Willems,et al.  The context-tree weighting method: basic properties , 1995, IEEE Trans. Inf. Theory.

[54]  Ofer Zeitouni,et al.  Asymptotic filtering for finite state Markov chains , 1996 .

[55]  Tsachy Weissman,et al.  Twofold universal prediction schemes for achieving the finite-state predictability of a noisy individual binary sequence , 2001, IEEE Trans. Inf. Theory.

[56]  Tsachy Weissman,et al.  Channel decoding of systematically encoded unknown redundant sources , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[57]  F. R. Gantmakher.  The Theory of Matrices , 1984 .

[58]  Cun-Hui Zhang,et al.  Compound decision theory and empirical bayes methods , 2003 .

[59]  J. V. Ryzin,et al.  The Sequential Compound Decision Problem with $m \times n$ Finite Loss Matrix , 1966 .

[60]  Pierre Soille,et al.  Morphological Image Analysis: Principles and Applications , 2003 .

[61]  S. Vardeman.  Admissible solutions of k-extended finite state set and sequence compound decision problems , 1980 .

[62]  E. Ordentlich,et al.  Inequalities for the L1 Deviation of the Empirical Distribution , 2003 .

[63]  Jaakko Astola,et al.  Classification and feature gene selection using the normalized maximum likelihood model for discrete regression , 2003, Signal Process..

[64]  Jorma Rissanen,et al.  MDL Denoising , 2000, IEEE Trans. Inf. Theory.

[65]  Tsachy Weissman,et al.  Universal prediction of random binary sequences in a noisy environment , 2004 .

[66]  J. M. Bilbao,et al.  Contributions to the Theory of Games , 2005 .

[67]  Jude W. Shavlik,et al.  Case-Based Reasoning with Noisy Case Boundaries: An Application in Molecular Biology , 1990 .

[68]  J. L. Devore.  A note on the observation of a Markov source through a noisy channel (Corresp.) , 1974, IEEE Trans. Inf. Theory.

[69]  Tsachy Weissman,et al.  Universal minimax binary image denoising under channel uncertainty , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[70]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[71]  Norbert Wiener,et al.  Extrapolation, Interpolation, and Smoothing of Stationary Time Series , 1964 .

[72]  I. Johnstone,et al.  Wavelet Shrinkage: Asymptopia? , 1995 .

[73]  Tsachy Weissman,et al.  Universal denoising for the finite-input general-output channel , 2005, IEEE Transactions on Information Theory.

[74]  Tsachy Weissman,et al.  Universally attainable error exponents for rate-distortion coding of noisy sources , 2003, IEEE Transactions on Information Theory.

[75]  M.J. Weinberger,et al.  Lossless compression of continuous-tone images , 2000, Proceedings of the IEEE.

[76]  J. Rissanen,et al.  Normalized Maximum Likelihood Models for Boolean Regression with Application to Prediction and Classification in Genomics , 2003 .

[77]  Tsachy Weissman.  Universally Attainable Error-Exponents for Rate-Constrained Denoising of Noisy Sources , 2002 .

[78]  M. Johns.  Two-action compound decision problems , 1967 .

[79]  Abraham Lempel,et al.  Compression of individual sequences via variable-rate coding , 1978, IEEE Trans. Inf. Theory.

[80]  Jude Shavlik,et al.  Finding Genes by Case-Based Reasoning in the Presence of Noisy Case Boundaries * , 1991 .

[81]  E. Samuel.  Convergence of the Losses of Certain Decision Rules for the Sequential Compound Decision Problem , 1964 .

[82]  Tsachy Weissman,et al.  On the optimality of symbol-by-symbol filtering and denoising , 2004, IEEE Transactions on Information Theory.