MDL Denoising Revisited

We refine and extend an earlier minimum description length (MDL) denoising criterion for wavelet-based denoising. We start by showing that the denoising problem can be reformulated as a clustering problem, where the goal is to obtain separate clusters for informative and noninformative wavelet coefficients, respectively. This suggests two refinements, adding a code-length for the model index, and extending the model in order to account for subband-dependent coefficient distributions. A third refinement is the derivation of soft thresholding inspired by predictive universal coding with weighted mixtures. We propose a practical method incorporating all three refinements, which is shown to achieve good performance and robustness in denoising both artificial and natural signals.

[1]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[2]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[3]  T. J. Mitchell,et al.  Bayesian Variable Selection in Linear Regression , 1988 .

[4]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[6]  Ronald A. DeVore,et al.  Image compression through wavelet transform coding , 1992, IEEE Trans. Inf. Theory.

[7]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[8]  D. L. Donoho,et al.  Ideal spacial adaptation via wavelet shrinkage , 1994 .

[9]  Naoki Saito,et al.  Simultaneous noise suppression and signal compression using a library of orthonormal bases and the minimum-description-length criterion , 1994, Defense, Security, and Sensing.

[10]  Balas K. Natarajan.  Filtering random noise from deterministic signals via data compression , 1995, IEEE Trans. Signal Process..

[11]  Frans M. J. Willems,et al.  The context-tree weighting method: basic properties , 1995, IEEE Trans. Inf. Theory.

[12]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[13]  Benjamin Belzer,et al.  Wavelet filter evaluation for image compression , 1995, IEEE Trans. Image Process..

[14]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[15]  Anestis Antoniadis,et al.  Model selection using wavelet decomposition and applications , 1997 .

[16]  E. George,et al.  APPROACHES FOR BAYESIAN VARIABLE SELECTION , 1997 .

[17]  H. Chipman,et al.  Adaptive Bayesian Wavelet Shrinkage , 1997 .

[18]  B. Vidakovic.  Nonlinear wavelet shrinkage with Bayes rules and Bayes factors , 1998 .

[19]  B. Silverman,et al.  Wavelet thresholding via a Bayesian approach , 1998 .

[20]  S. Mallat.  A wavelet tour of signal processing , 1998 .

[21]  Martin J. Wainwright,et al.  Scale Mixtures of Gaussians and the Statistics of Natural Images , 1999, NIPS.

[22]  Pierre Moulin,et al.  Analysis of Multiresolution Image Denoising Schemes Using Generalized Gaussian and Complexity Priors , 1999, IEEE Trans. Inf. Theory.

[23]  Hamid Krim,et al.  Minimax Description Length for Signal Denoising and Optimized Representation , 1999, IEEE Trans. Inf. Theory.

[24]  Brani Vidakovic,et al.  A BAYESIAN DECISION THEORETIC APPROACH TO THE CHOICE OF THRESHOLDING PARAMETER , 1999 .

[25]  Martin Vetterli,et al.  Adaptive wavelet thresholding for image denoising and compression , 2000, IEEE Trans. Image Process..

[26]  Dean P. Foster,et al.  The Competitive Complexity Ratio , 2000 .

[27]  Andrew R. Barron,et al.  Asymptotic minimax regret for data compression, gambling, and prediction , 1997, IEEE Trans. Inf. Theory.

[28]  Bin Yu,et al.  Wavelet thresholding via MDL for natural images , 2000, IEEE Trans. Inf. Theory.

[29]  I. Csiszár,et al.  The consistency of the BIC Markov order estimator , 2000 .

[30]  Jorma Rissanen,et al.  MDL Denoising , 2000, IEEE Trans. Inf. Theory.

[31]  Pierre Moulin,et al.  Complexity-regularized image denoising , 2001, IEEE Trans. Image Process..

[32]  Jorma Rissanen,et al.  Strong optimality of the normalized ML models as universal codes and information in data , 2001, IEEE Trans. Inf. Theory.

[33]  Jorma Rissanen,et al.  Lectures on Statistical Modeling Theory , 2002 .

[34]  Martin J. Wainwright,et al.  Image denoising using scale mixtures of Gaussians in the wavelet domain , 2003, IEEE Trans. Image Process..

[35]  Daniel J. Navarro,et al.  A Note on the Applied Use of MDL Approximations , 2004, Neural Computation.

[36]  Nikolai K. Vereshchagin,et al.  Kolmogorov's structure functions and model selection , 2002, IEEE Transactions on Information Theory.

[37]  Feng Liang,et al.  Exact minimax strategies for predictive density estimation, data compression, and model selection , 2002, IEEE Transactions on Information Theory.

[38]  Timo Miettinen,et al.  Robust denoising of electrophoresis and mass spectrometry signals with minimum description length principle , 2004, FEBS letters.

[39]  Peter Grünwald,et al.  A tutorial introduction to the minimum description length principle , 2004, ArXiv.

[40]  Y. Shtarkov,et al.  The context-tree weighting method: basic properties , 1995, IEEE Trans. Inf. Theory.

[41]  Jorma Rissanen,et al.  An MDL Framework for Data Clustering , 2005 .

[42]  Henry Tirri,et al.  On the Behavior of MDL Denoising , 2005, AISTATS.

[43]  Steven de Rooij,et al.  An Empirical Study of MDL Model Selection with Infinite Parametric Complexity , 2005, ArXiv.

[44]  Jorma Rissanen,et al.  Information and Complexity in Statistical Modeling , 2006, ITW.

[45]  Jukka Heikkonen,et al.  Minimum Description Length Denoising With Histogram Models , 2006, IEEE Transactions on Signal Processing.

[46]  Peter Grünwald,et al.  Accumulative prediction error and the selection of time series models , 2006 .

[47]  Jay I. Myung,et al.  Model selection by Normalized Maximum Likelihood , 2006 .

[48]  K. Saito,et al.  Tooth shape reconstruction from ct images using spline Curves , 2007, 2007 International Conference on Wavelet Analysis and Pattern Recognition.

[49]  P. Grünwald.  The Minimum Description Length Principle (Adaptive Computation and Machine Learning) , 2007 .

[50]  Stphane Mallat,et al.  A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way , 2008 .

[51]  Jorma Rissanen,et al.  Minimum Description Length Principle , 2010, Encyclopedia of Machine Learning.

[52]  Dean P. Foster,et al.  The Contribution of Parameters to Stochastic Complexity , 2022 .