An efficient approach for computing optimal low-rank regularized inverse matrices

Standard regularization methods that are used to compute solutions to ill-posed inverse problems require knowledge of the forward model. In many real-life applications, the forward model is not known, but training data is readily available. In this paper, we develop a new framework that uses training data, as a substitute for knowledge of the forward model, to compute an optimal low-rank regularized inverse matrix directly, allowing for very fast computation of a regularized solution. We consider a statistical framework based on Bayes and empirical Bayes risk minimization to analyze theoretical properties of the problem. We propose an efficient rank update approach for computing an optimal low-rank regularized inverse matrix for various error measures. Numerical experiments demonstrate the benefits and potential applications of our approach to problems in signal and image processing.

[1]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[2]  T. Louis,et al.  BAYES AND EMPIRICAL BAYES METHODS FOR DATA ANALYSIS , 1996, Stat. Comput..

[3]  Juliane Hahn.  Inverse Problems Mathematical And Analytical Techniques With Applications To Engineering , 2016 .

[4]  Dianne P. O'Leary,et al.  Designing Optimal Spectral Filters for Inverse Problems , 2011, SIAM J. Sci. Comput..

[5]  Dianne P. O'Leary,et al.  Deblurring Images: Matrices, Spectra and Filtering , 2006, J. Electronic Imaging.

[6]  Gene H. Golub,et al.  Matrix computations , 1983 .

[7]  K. Egiazarian,et al.  Blind image deconvolution , 2007 .

[8]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[9]  Hsuan-Tien Lin,et al.  Learning From Data , 2012 .

[10]  Francisco Duarte Moura Neto,et al.  An Introduction to Inverse Problems with Applications , 2012 .

[11]  J. Hadamard,et al.  Lectures on Cauchy's Problem in Linear Partial Differential Equations , 1924 .

[12]  Albert Tarantola,et al.  Inverse problem theory - and methods for model parameter estimation , 2004 .

[13]  Per Christian Hansen,et al.  Rank-Deficient and Discrete Ill-Posed Problems , 1996 .

[14]  D. Sondermann.  Best approximate solutions to matrix equations under rank restrictions , 1986 .

[15]  Dianne P. O'Leary,et al.  Optimal regularized low rank inverse approximation , 2015 .

[16]  Alan J. Lee,et al.  Linear Regression Analysis: Seber/Linear , 2003 .

[17]  Philip E. Gill,et al.  Practical optimization , 1981 .

[18]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[19]  George A. F. Seber,et al.  Linear regression analysis , 1977 .

[20]  Bradley P. Carlin,et al.  BAYES AND EMPIRICAL BAYES METHODS FOR DATA ANALYSIS , 1996, Stat. Comput..

[21]  Julianne Chung,et al.  Computing optimal low-rank matrix approximations for image processing , 2013, 2013 Asilomar Conference on Signals, Systems and Computers.

[22]  A. Kirsch.  An Introduction to the Mathematical Theory of Inverse Problems , 1996, Applied Mathematical Sciences.

[23]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[24]  C. Eckart,et al.  The approximation of one matrix by another of lower rank , 1936 .

[25]  C. Vogel.  Computational Methods for Inverse Problems , 1987 .

[26]  Pierre Baldi,et al.  Bioinformatics - the machine learning approach (2. ed.) , 2000 .

[27]  Anatoli Torokhti,et al.  Generalized Rank-Constrained Matrix Approximations , 2007, SIAM J. Matrix Anal. Appl..

[28]  Jianhong Shen,et al.  Deblurring images: Matrices, spectra, and filtering , 2007, Math. Comput..

[29]  Alexander Shapiro,et al.  Lectures on Stochastic Programming: Modeling and Theory , 2009 .

[30]  Clifford H. Thurber,et al.  Parameter estimation and inverse problems , 2005 .

[31]  Simon Parsons,et al.  Bioinformatics: The Machine Learning Approach by P. Baldi and S. Brunak, 2nd edn, MIT Press, 452 pp., $60.00, ISBN 0-262-02506-X , 2004, The Knowledge Engineering Review.

[32]  Jean-Luc Starck,et al.  Deconvolution and Blind Deconvolution in Astronomy , 2007 .

[33]  H. Engl,et al.  Regularization of Inverse Problems , 1996 .

[34]  Dianne P. O'Leary,et al.  Optimal Filters from Calibration Data for Image Deconvolution with Data Acquisition Error , 2012, Journal of Mathematical Imaging and Vision.

[35]  E. Schmidt.  Zur Theorie der linearen und nichtlinearen Integralgleichungen , 1907 .

[36]  A. Ramm,et al.  Inverse Problems: Mathematical and Analytical Techniques with Applications to Engineering , 2004 .

[37]  G. W. Stewart,et al.  Matrix Algorithms: Volume 1, Basic Decompositions , 1998 .

[38]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.