Multi-Penalty Regularization for Detecting Relevant Variables

In this paper, we propose a new method for detecting relevant variables from a priori given high-dimensional data under the assumption that input-output relation is described by a nonlinear function depending on a few variables. The method is based on the inspection of the behavior of discrepancies of a multi-penalty regularization with a component-wise penalization for small and large values of regularization parameters. We provide a justification of the proposed method under a certain condition on sampling operators. The effectiveness of the method is demonstrated in an example with simulated data and in the reconstruction of a gene regulatory network. In the latter example, the obtained results provide clear evidence of the competitiveness of the proposed method with respect to the state-of-the-art approaches.

[1]  Holger Rauhut,et al.  A Mathematical Introduction to Compressive Sensing , 2013, Applied and Numerical Harmonic Analysis.

[2]  Lorenzo Rosasco,et al.  Nonparametric sparsity and regularization , 2012, J. Mach. Learn. Res..

[3]  Robert Hable,et al.  Consistency of support vector machines using additive kernels for additive models , 2012, Comput. Stat. Data Anal..

[4]  Francis R. Bach,et al.  Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning , 2008, NIPS.

[5]  K. Hlavácková-Schindler,et al.  Causality detection based on information-theoretic approaches in time series analysis , 2007 .

[6]  C. Granger.  Investigating causal relations by econometric models and cross-spectral methods , 1969 .

[7]  Ronny Ramlau,et al.  A Tikhonov-based projection iteration for nonlinear Ill-posed problems with sparsity constraints , 2006, Numerische Mathematik.

[8]  Ali Shojaie,et al.  Discovering graphical Granger causality using the truncating lasso penalty , 2010, Bioinform..

[9]  Naoki Abe,et al.  Grouped graphical Granger modeling for gene expression regulatory networks discovery , 2009, Bioinform..

[10]  慧 廣瀬.  A Mathematical Introduction to Compressive Sensing , 2015 .

[11]  Bernhard Schölkopf,et al.  A Generalized Representer Theorem , 2001, COLT/EuroCOLT.

[12]  Valeriya Naumova,et al.  Multi-penalty regularization with a component-wise penalization , 2013 .

[13]  V. Koltchinskii,et al.  SPARSITY IN MULTIPLE KERNEL LEARNING , 2010, 1211.2998.

[14]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[15]  O. Scherzer,et al.  Sparse regularization with lq penalty term , 2008, 0806.3222.

[16]  C. Granger.  Investigating Causal Relations by Econometric Models and Cross-Spectral Methods , 1969 .

[17]  Stefan Kindermann,et al.  On the convergence of the quasioptimality criterion for (iterated) Tikhonov regularization , 2008 .

[18]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[19]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[20]  C. Ball,et al.  Identification of genes periodically expressed in the human cell cycle and their expression in tumors. , 2002, Molecular biology of the cell.

[21]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[22]  Michael I. Jordan,et al.  Multiple kernel learning, conic duality, and the SMO algorithm , 2004, ICML.

[23]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[24]  G. Toffolo,et al.  CNET: an algorithm for Reverse Engineering of Causal Gene Networks , 2008 .

[25]  Markus Reiss,et al.  Regularization independent of the noise level: an analysis of quasi-optimality , 2007, 0710.1045.

[26]  Christian Berg,et al.  Positive Definite Functions and Moment Functions , 1984 .

[27]  Katerina Hlavácková-Schindler,et al.  Graphical Lasso Granger Method with 2-Levels-Thresholding for Recovering Causality Networks , 2013, System Modelling and Optimization.

[28]  R. Tibshirani,et al.  Generalized Additive Models , 1986 .

[29]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[30]  Yan Liu,et al.  Temporal causal modeling with graphical granger methods , 2007, KDD '07.

[31]  G. Wahba,et al.  A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines , 1970 .

[32]  P. Zhao,et al.  The composite absolute penalties family for grouped and hierarchical variable selection , 2009, 0909.0411.

[33]  H. Engl,et al.  Inverse problems in systems biology , 2009 .

[34]  Kai Ming Ting,et al.  Precision and Recall , 2017, Encyclopedia of Machine Learning and Data Mining.

[35]  O. Scherzer,et al.  FA ] 2 9 Ju l 2 00 8 Sparse Regularization with l q Penalty Term , 2008 .

[36]  A. Tikhonov,et al.  Use of the regularization method in non-linear problems , 1965 .

[37]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[38]  J. Lafferty,et al.  Sparse additive models , 2007, 0711.4555.

[39]  C. Berg,et al.  Harmonic Analysis on Semigroups , 1984 .

[40]  Li Li,et al.  Discovery of time-delayed gene regulatory networks based on temporal gene expression profiling , 2006, BMC Bioinformatics.