Automated Spectral Smoothing with Spatially Adaptive Penalized Least Squares

A variety of data smoothing techniques exist to address the issue of noise in spectroscopic data. The vast majority, however, require parameter specification by a knowledgeable user, which is typically accomplished by trial and error. In most situations, optimized parameters represent a compromise between noise reduction and signal preservation. In this work, we demonstrate a nonparametric regression approach to spectral smoothing using a spatially adaptive penalized least squares (SAPLS) approach. An iterative optimization procedure is employed that permits gradual flexibility in the smooth fit when statistically significant trends based on multiscale statistics assuming white Gaussian noise are detected. With an estimate of the noise level in the spectrum the procedure is fully automatic with a specified confidence level for the statistics. Potential application to the heteroscedastic noise case is also demonstrated. Performance was assessed in simulations conducted on several synthetic spectra using traditional error measures as well as comparisons of local extrema in the resulting smoothed signals to those in the true spectra. For the simulated spectra, a best case comparison with the Savitzky–Golay smoothing via an exhaustive parameter search was performed while the SAPLS method was assessed for automated application. The application to several dissimilar experimentally obtained Raman spectra is also presented.

[1]  Arne Kovac,et al.  Extensions of Smoothing via Taut Strings , 2008, 0803.2931.

[2]  P. Schaeffer,et al.  Catabolic repression of bacterial sporulation. , 1965, Proceedings of the National Academy of Sciences of the United States of America.

[3]  H. Georg Schulze,et al.  Two-Point Maximum Entropy Noise Discrimination in Spectra over a Range of Baseline Offsets and Signal-to-Noise Ratios , 2007, Applied spectroscopy.

[4]  Łukasz Komsta,et al.  A comparative study on several algorithms for denoising of thin layer densitograms. , 2009, Analytica chimica acta.

[5]  A. Kovac,et al.  Smooth functions and local extreme values , 2007, Comput. Stat. Data Anal..

[6]  Irène Gijbels,et al.  A study of variable bandwidth selection for local polynomial regression , 1996 .

[7]  R. Bonner,et al.  Application of wavelet transforms to experimental spectra : Smoothing, denoising, and data set compression , 1997 .

[8]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[9]  A B Novikov,et al.  Automatic correction of continuum background in laser-induced breakdown and Raman spectrometry. , 2003, Applied spectroscopy.

[10]  Prakash N. Patil,et al.  On the Choice of Smoothing Parameter, Threshold and Truncation in Nonparametric Regression by Non-linear Wavelet Methods , 1996 .

[11]  D. Mergel,et al.  Residual-based localization and quantification of peaks in X-ray diffractograms , 2007 .

[12]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[13]  B. Silverman,et al.  Nonparametric Regression and Generalized Linear Models: A roughness penalty approach , 1993 .

[14]  G. Wahba.  A Comparison of GCV and GML for Choosing the Smoothing Parameter in the Generalized Spline Smoothing Problem , 1985 .

[15]  Dor Ben-Amotz,et al.  Stripping of Cosmic Spike Spectral Artifacts Using a New Upper-Bound Spectrum Algorithm , 2001 .

[16]  Jianqing Fan,et al.  Local polynomial kernel regression for generalized linear models and quasi-likelihood functions , 1995 .

[17]  E. Prescott,et al.  Postwar U.S. Business Cycles: An Empirical Investigation , 1997 .

[18]  P. Davies,et al.  Approximating data with weighted smoothing splines , 2007, 0712.1692.

[19]  M. Grasserbauer,et al.  Wavelet denoising of Gaussian peaks: A comparative study , 1996 .

[20]  Arne Kovac,et al.  Local extremes, runs, strings and multiresolution - Rejoinder , 2001 .

[21]  E. Parzen,et al.  Data dependent wavelet thresholding in nonparametric regression with change-point applications , 1996 .

[22]  Andre Ivanov,et al.  Fully Automated High-Performance Signal-to-Noise Ratio Enhancement Based on an Iterative Three-Point Zero-Order Savitzky—Golay Filter , 2008, Applied spectroscopy.

[23]  Edmund Taylor Whittaker,et al.  VIII.—On the Theory of Graduation , 1925 .

[24]  P. A. Gorry.  General least-squares smoothing and differentiation by the convolution (Savitzky-Golay) method , 1990 .

[25]  Arne Kovac,et al.  Robust nonparametric regression and modality , 2003 .

[26]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[27]  Angelika Rohde.  Adaptive goodness-of-fit tests based on signed ranks , 2008 .

[28]  P. Schoenmakers,et al.  Automatic selection of optimal Savitzky-Golay smoothing. , 2006, Analytical chemistry.

[29]  H Georg Schulze,et al.  Automated Estimation of White Gaussian Noise Level in a Spectrum with or without Spike Noise Using a Spectral Shifting Technique , 2006, Applied spectroscopy.

[30]  P. Davies,et al.  Nonparametric Regression, Confidence Regions and Regularization , 2007, 0711.0690.

[31]  Xiao-Ping Xu,et al.  Efficient automatic noise reduction of electron tomographic reconstructions based on iterative median filtering. , 2007, Journal of structural biology.

[32]  Lutz Dümbgen,et al.  Application of local rank tests to nonparametric regression , 2002 .

[33]  Andrew Jirasek,et al.  Investigation of Selected Baseline Removal Techniques as Candidates for Automated Implementation , 2005, Applied spectroscopy.

[34]  Gregory W. Auner,et al.  A robust method for automated background subtraction of tissue fluorescence , 2007 .

[35]  Hideo Takeuchi,et al.  Simple and Efficient Method to Eliminate Spike Noise from Spectra Recorded on Charge-Coupled Device Detectors , 1993 .

[36]  A. Savitzky,et al.  Smoothing and Differentiation of Data by Simplified Least Squares Procedures. , 1964 .

[37]  G. Wahba,et al.  Hybrid Adaptive Splines , 1997 .

[38]  A. Mahadevan-Jansen,et al.  Automated Method for Subtraction of Fluorescence from Biological Raman Spectra , 2003, Applied spectroscopy.

[39]  Christopher Holmes,et al.  Spatially adaptive smoothing splines , 2006 .

[40]  Pavel Matějka,et al.  Noise reduction in Raman spectra: Finite impulse response filtration versus Savitzky–Golay smoothing , 2007 .

[41]  P. Eilers.  A perfect smoother. , 2003, Analytical chemistry.

[42]  A. Y. Chikishev,et al.  Optimization of the Rolling-Circle Filter for Raman Background Subtraction , 2006, Applied spectroscopy.

[43]  V. Spokoiny,et al.  Multiscale testing of qualitative hypotheses , 2001 .

[44]  Jianqing Fan,et al.  Data‐Driven Bandwidth Selection in Local Polynomial Fitting: Variable Bandwidth and Spatial Adaptation , 1995 .

[45]  D. Cox.  Nonparametric Regression and Generalized Linear Models: A roughness penalty approach , 1993 .

[46]  P. Davies,et al.  Local Extremes, Runs, Strings and Multiresolution , 2001 .

[47]  Norbert Michael Mayer,et al.  A multiscale polynomial filter for adaptive smoothing , 2007, Digit. Signal Process..

[48]  Manuel Martín-Pastor,et al.  A new general-purpose fully automatic baseline-correction procedure for 1D and 2D NMR data. , 2006, Journal of magnetic resonance.

[49]  Andre Ivanov,et al.  Chi-Squared-Based Filters for High-Fidelity Signal-to-Noise Ratio Enhancement of Spectra , 2008, Applied spectroscopy.

[50]  Yukihiro Ozaki,et al.  Practical Algorithm for Reducing Convex Spike Noises on a Spectrum , 2003, Applied spectroscopy.

[51]  Satoshi Miyata,et al.  Adaptive Free-Knot Splines , 2003 .

[52]  Paul H. C. Eilers,et al.  Flexible smoothing with B-splines and penalties , 1996 .

[53]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.