Residual periodograms for choosing regularization parameters for ill-posed problems

Consider an ill-posed problem transformed if necessary so that the errors in the data are independent identically normally distributed with mean zero and variance 1. We survey regularization and parameter selection from a linear algebra and statistics viewpoint and compare the statistical distributions of regularized estimates of the solution and the residual. We discuss methods for choosing a regularization parameter in order to assure that the residual for the model is statistically plausible. Ideally, as proposed by Rust (1998 Tech. Rep. NISTIR 6131, 2000 Comput. Sci. Stat. 32 333–47 ), the results of candidate parameter choices should be evaluated by plotting the resulting residual along with its periodogram and its cumulative periodogram, but sometimes an automated choice is needed. We evaluate a method for choosing the regularization parameter that makes the residuals as close as possible to white noise, using a diagnostic test based on the periodogram. We compare this method with standard techniques such as the discrepancy principle, the L-curve and generalized cross validation, showing that it performs better on two new test problems as well as a variety of standard problems.

[1]  V. Hutson.  Integral Equations , 1967, Nature.

[2]  G. Stewart,et al.  Rank degeneracy and least squares problems , 1976 .

[3]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[4]  F. Agterberg.  Introduction to Mathematics of Inversion in Remote Sensing and Indirect Measurements , 1979 .

[5]  Wolfgang Osten,et al.  Introduction to Inverse Problems in Imaging , 1999 .

[6]  P. Hansen.  Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion , 1987 .

[7]  Anja Vogler,et al.  An Introduction to Multivariate Statistical Analysis , 2004 .

[8]  A Tikhonov,et al.  Solution of Incorrectly Formulated Problems and the Regularization Method , 1963 .

[9]  Mario Bertero,et al.  Introduction to Inverse Problems in Imaging , 1998 .

[10]  Dianne P. O'Leary,et al.  The Use of the L-Curve in the Regularization of Discrete Ill-Posed Problems , 1993, SIAM J. Sci. Comput..

[11]  G. Wahba.  Practical Approximate Solutions to Linear Operator Equations When the Data are Noisy , 1977 .

[12]  Bert W. Rust,et al.  Parameter Selection for Constrained Solutions to III-Posed Problems , 2000 .

[13]  B. Rust.  Truncating the Singular Value Decomposition for Ill-Posed Problems , 1998 .

[14]  W. R. Burrus,et al.  Fast-neutron spectroscopy with thick organic scintillators , 1969 .

[15]  Per Christian Hansen,et al.  Analysis of Discrete Ill-Posed Problems by Means of the L-Curve , 1992, SIAM Rev..

[16]  Chris Chatfield,et al.  Introduction to Statistical Time Series. , 1976 .

[17]  David L. Phillips,et al.  A Technique for the Numerical Solution of Certain Integral Equations of the First Kind , 1962, JACM.

[18]  A. Tikhonov,et al.  Numerical Methods for the Solution of Ill-Posed Problems , 1995 .

[19]  Misha Elena Kilmer,et al.  Choosing Regularization Parameters in Iterative Methods for Ill-Posed Problems , 2000, SIAM J. Matrix Anal. Appl..

[20]  P. Hoel,et al.  Introduction to Mathematical Statistics. Second Edition. , 1955 .

[21]  A. Morelli.  Inverse Problem Theory , 2010 .

[22]  H. Engl,et al.  Regularization of Inverse Problems , 1996 .

[23]  R. Hanson.  A Numerical Method for Solving Fredholm Integral Equations of the First Kind Using Singular Values , 1971 .

[24]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[25]  Per Christian Hansen,et al.  REGULARIZATION TOOLS: A Matlab package for analysis and solution of discrete ill-posed problems , 1994, Numerical Algorithms.

[26]  T. W. Anderson,et al.  An Introduction to Multivariate Statistical Analysis , 1959 .

[27]  V. Morozov.  On the solution of functional equations by the method of regularization , 1966 .

[28]  W. R. Burrus,et al.  MATHEMATICAL PROGRAMMING AND THE NUMERICAL SOLUTION OF LINEAR EQUATIONS. , 1976 .

[29]  P. Hansen,et al.  Exploiting Residual Information in the Parameter Choice for Discrete Ill-Posed Problems , 2006 .

[30]  DAVID G. KENDALL,et al.  Introduction to Mathematical Statistics , 1947, Nature.

[31]  G. Wing,et al.  A Primer on Integral Equations of the First Kind: The Problem of Deconvolution and Unfolding , 1987 .

[32]  A. E. Hoerl,et al.  Ridge Regression: Applications to Nonorthogonal Problems , 1970 .

[33]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .

[34]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[35]  S. Twomey,et al.  On the Numerical Solution of Fredholm Integral Equations of the First Kind by the Inversion of the Linear System Produced by Quadrature , 1963, JACM.

[36]  S. Twomey.  Introduction to the Mathematics of Inversion in Remote Sensing and Indirect Measurements , 1997 .

[37]  W. R. Burrus,et al.  CALIBRATION OF AN ORGANIC SCINTILLATOR FOR NEUTRON SPECTROMETRY. , 1968 .

[38]  Harvey Thomas Banks,et al.  Standard errors and confidence intervals in inverse problems: sensitivity and associated pitfalls , 2007 .

[39]  Gene H. Golub,et al.  Calculating the singular values and pseudo-inverse of a matrix , 2007, Milestones in Matrix Computation.

[40]  T. W. Anderson.  An Introduction to Multivariate Statistical Analysis , 1959 .

[41]  Jodi Mead,et al.  Parameter estimation: A new approach to weighting a priori information , 2007 .

[42]  Joseph F. McGrath,et al.  A Primer on Integral Equations of the First Kind: The Problem of Deconvolution and Unfolding (G. Milton Wing with the assistance of John D. Zahrt) , 1993, SIAM Rev..