Multi-parameter regularization and its numerical realization

In this paper we propose and analyse a choice of parameters in the multi-parameter regularization of Tikhonov type. A modified discrepancy principle is presented within the multi-parameter regularization framework. An order optimal error bound is obtained under the standard smoothness assumptions. We also propose a numerical realization of the multi-parameter discrepancy principle based on the model function approximation. Numerical experiments on a series of test problems support theoretical results. Finally we show how the proposed approach can be successfully implemented in Laplacian Regularized Least Squares for learning from labeled and unlabeled examples.

[1]  L PhillipsDavid.  A Technique for the Numerical Solution of Certain Integral Equations of the First Kind , 1962 .

[2]  Bernd Hofmann,et al.  A multi-parameter regularization approach for estimating parameters in jump diffusion processes , 2006 .

[3]  Olha Ivanyshyn,et al.  Optimal regularization with two interdependent regularization parameters , 2007 .

[4]  W. Hemmerle.  An Explicit Solution for Generalized Ridge Regression , 1975 .

[5]  Sergei V. Pereverzev,et al.  A MODEL FUNCTION METHOD IN TOTAL LEAST SQUARES , 2008 .

[6]  K. Kunisch,et al.  Iterative choices of regularization parameters in linear inverse problems , 1998 .

[7]  Yu. I. Petunin,et al.  SCALES OF BANACH SPACES , 1966 .

[8]  Jun Zou,et al.  An improved model function method for choosing regularization parameters in linear inverse problems , 2002 .

[9]  T. Poggio,et al.  The Mathematics of Learning: Dealing with Data , 2005, 2005 International Conference on Neural Networks and Brain.

[10]  H. Engl,et al.  Johann Radon Institute for Computational and Applied Mathematics , 2004 .

[11]  William J. Hemmerle,et al.  Explicit and Constrained Generalized Ridge Estimation , 1978 .

[12]  Bernd Hofmann,et al.  Analysis of Profile Functions for General Linear Regularization Methods , 2007, SIAM J. Numer. Anal..

[13]  Bernard A. Mair,et al.  Tikhonov regularization for finitely and infinitely smoothing operators , 1994 .

[14]  Sergei V. Pereverzev,et al.  On the generalized discrepancy principle for Tikhonov regularization in Hilbert scales , 2010 .

[15]  U. TAUTENHAHN,et al.  Dual Regularized Total Least Squares And Multi-Parameter Regularization , 2008 .

[16]  V. Morozov.  On the solution of functional equations by the method of regularization , 1966 .

[17]  Olivier Chapelle,et al.  Training a Support Vector Machine in the Primal , 2007, Neural Computation.

[18]  Barbara Kaltenbacher,et al.  Iterative Regularization Methods for Nonlinear Ill-Posed Problems , 2008, Radon Series on Computational and Applied Mathematics.

[19]  Lorenzo Rosasco,et al.  Adaptive Kernel Methods Using the Balancing Principle , 2010, Found. Comput. Math..

[20]  Peter Mathé.  Saturation of Regularization Methods for Linear Ill-Posed Problems in Hilbert Spaces , 2004, SIAM J. Numer. Anal..

[21]  F. Natterer.  Error bounds for tikhonov regularization in hilbert scales , 1984 .

[22]  Franz Schreier,et al.  Iterative regularization methods for nonlinear problems , 2010 .

[23]  Sergei V. Pereverzev,et al.  Regularization in Hilbert scales under general smoothing conditions , 2005 .

[24]  Rob S. MacLeod,et al.  Inverse electrocardiography by simultaneous imposition of multiple constraints , 1999, IEEE Transactions on Biomedical Engineering.

[25]  David L. Phillips,et al.  A Technique for the Numerical Solution of Certain Integral Equations of the First Kind , 1962, JACM.

[26]  E. Miller,et al.  Efficient determination of multiple regularization parameters in a generalized L-curve framework , 2002 .

[27]  Peiliang Xu,et al.  Multiple Parameter Regularization: Numerical Solutions and Applications to the Determination of Geopotential from Precise Satellite Orbits , 2006 .

[28]  Bernd Hofmann,et al.  Convergence rates for Tikhonov regularization from different kinds of smoothness conditions , 2006 .

[29]  H. Engl,et al.  Regularization of Inverse Problems , 1996 .

[30]  P. Mathé,et al.  Geometry of linear ill-posed problems in variable Hilbert scales Inverse Problems 19 789-803 , 2003 .

[31]  Lorenzo Rosasco,et al.  On regularization algorithms in learning theory , 2007, J. Complex..

[32]  Claude Brezinski,et al.  Multi-parameter regularization techniques for ill-conditioned linear systems , 2003, Numerische Mathematik.

[33]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[34]  D. Lorenz,et al.  Elastic-net regularization: error estimates and active set methods , 2009, 0905.0796.

[35]  Per Christian Hansen,et al.  REGULARIZATION TOOLS: A Matlab package for analysis and solution of discrete ill-posed problems , 1994, Numerical Algorithms.

[36]  D. Barry.  Nonparametric Bayesian Regression , 1986 .

[37]  Zhongying,et al.  MULTI-PARAMETER TIKHONOV REGULARIZATION FOR LINEAR ILL-POSED OPERATOR EQUATIONS , 2008 .

[38]  Shuai Lu,et al.  Sparse recovery by the standard Tikhonov method , 2009, Numerische Mathematik.

[39]  Shuai Lu,et al.  Regularized Total Least Squares: Computational Aspects and Error Bounds , 2009, SIAM J. Matrix Anal. Appl..

[40]  Charles A. Micchelli,et al.  Learning the Kernel Function via Regularization , 2005, J. Mach. Learn. Res..