Safe Screening for Sparse Regression with the Kullback-Leibler Divergence

Safe screening rules are powerful tools to accelerate iterative solvers in sparse regression problems. They allow early identification of inactive coordinates (i.e., those not belonging to the support of the solution) which can thus be screened out in the course of iterations. In this paper, we extend the GAP Safe screening rule to the ℓ1-regularized Kullback-Leibler divergence which does not fulfil the regularity assumptions made in previous works. The proposed approach is experimentally validated on synthetic and real count data sets.

[1]  Thierry Bertin-Mahieux,et al.  The Million Song Dataset , 2011, ISMIR.

[2]  B. Melamed,et al.  Traffic modeling for telecommunications networks , 1994, IEEE Communications Magazine.

[3]  L. Lucy.  An iterative technique for the rectification of observed distributions , 1974 .

[4]  Jieping Ye,et al.  Two-Layer Feature Reduction for Sparse-Group Lasso via Decomposition of Convex Sets , 2014, NIPS.

[5]  Rebecca Willett,et al.  This is SPIRAL-TAP: Sparse Poisson Intensity Reconstruction ALgorithms—Theory and Practice , 2010, IEEE Transactions on Image Processing.

[6]  Jérôme Idier,et al.  Algorithms for Nonnegative Matrix Factorization with the β-Divergence , 2010, Neural Computation.

[7]  William H. Richardson,et al.  Bayesian-Based Iterative Method of Image Restoration , 1972 .

[8]  Alexandre Gramfort,et al.  Gap Safe screening rules for sparsity enforcing penalties , 2016, J. Mach. Learn. Res..

[9]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[10]  Nelly Pustelnik,et al.  Hybrid regularization for data restoration in the presence of Poisson noise , 2009, 2009 17th European Signal Processing Conference.

[11]  Laurent El Ghaoui,et al.  Safe Feature Elimination for the LASSO and Sparse Supervised Learning Problems , 2010, 1009.4219.

[12]  Jieping Ye,et al.  Fused Lasso Screening Rules via the Monotonicity of Subdifferentials , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Alexandre Gramfort,et al.  Mind the duality gap: safer rules for the Lasso , 2015, ICML.

[14]  Francis R. Bach,et al.  Primal-dual algorithms for non-negative matrix factorization with the Kullback-Leibler divergence , 2014, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[15]  Rémi Gribonval,et al.  Dynamic Screening: Accelerating First-Order Algorithms for the Lasso and Group-Lasso , 2014, IEEE Transactions on Signal Processing.

[16]  Jiayu Zhou,et al.  A Safe Screening Rule for Sparse Logistic Regression , 2013, NIPS.

[17]  D. Titterington.  On the Iterative Image Space Reconstruction Algorthm for ECT , 1987, IEEE Transactions on Medical Imaging.

[18]  Haesun Park,et al.  Fast bregman divergence NMF using taylor expansion and coordinate descent , 2012, KDD.

[19]  Giovanni Chierchia,et al.  Proximity Operators of Discrete Information Divergences , 2016, IEEE Transactions on Information Theory.

[20]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[21]  Inderjit S. Dhillon,et al.  Fast coordinate descent methods with variable selection for non-negative matrix factorization , 2011, KDD.