A Nearly-Linear Time Framework for Graph-Structured Sparsity

We introduce a framework for sparsity structures defined via graphs. Our approach is flexible and generalizes several previously studied sparsity models. Moreover, we provide efficient projection algorithms for our sparsity model that run in nearly-linear time. In the context of sparse recovery, we show that our framework achieves an information-theoretically optimal sample complexity for a wide range of parameters. We complement our theoretical analysis with experiments demonstrating that our algorithms also improve on prior work in practice.

[1]  Julien Mairal,et al.  Optimization with Sparsity-Inducing Penalties , 2011, Found. Trends Mach. Learn..

[2]  Jean-Philippe Vert,et al.  Group lasso with overlap and graph lasso , 2009, ICML '09.

[3]  Piotr Indyk,et al.  Approximation Algorithms for Model-Based Compressive Sensing , 2014, IEEE Transactions on Information Theory.

[4]  David Eisenstat,et al.  An efficient polynomial-time approximation scheme for Steiner forest in planar graphs , 2012, SODA.

[5]  Volkan Cevher,et al.  A totally unimodular view of structured sparsity , 2014, AISTATS.

[6]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[7]  Mohammad Taghi Hajiaghayi,et al.  Prize-collecting Steiner problems on planar graphs , 2011, SODA '11.

[8]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[9]  Carlos Eduardo Ferreira,et al.  A note on Johnson, Minkoff and Phillips' algorithm for the Prize-Collecting Steiner Tree Problem , 2010, ArXiv.

[10]  David S. Johnson,et al.  The prize collecting Steiner tree problem: theory and practice , 2000, SODA '00.

[11]  Axthonv G. Oettinger,et al.  IEEE Transactions on Information Theory , 1998 .

[12]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[13]  D. Balding,et al.  Structured Regularizers for High-Dimensional Problems : Statistical and Computational Issues , 2014 .

[14]  Richard Baraniuk,et al.  Recovery of Clustered Sparse Signals from Compressive Measurements , 2009 .

[15]  Robert D. Nowak,et al.  Universal Measurement Bounds for Structured Sparse Signal Recovery , 2012, AISTATS.

[16]  Junzhou Huang,et al.  Learning with structured sparsity , 2009, ICML '09.

[17]  Yonina C. Eldar,et al.  Structured Compressed Sensing: From Theory to Applications , 2011, IEEE Transactions on Signal Processing.

[18]  Piotr Indyk,et al.  Nearly Linear-Time Model-Based Compressive Sensing , 2014, ICALP.

[19]  Michael Jünger,et al.  New primal and dual matching heuristics , 1995, Algorithmica.

[20]  Aristides Gionis,et al.  ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2014, New York, NY, USA - August 24 - 27, 2014 , 2014 .

[21]  Eric P. Xing,et al.  Tree-Guided Group Lasso for Multi-Task Regression with Structured Sparsity , 2009, ICML.

[22]  A. Sayed,et al.  Foundations and Trends ® in Machine Learning > Vol 7 > Issue 4-5 Ordering Info About Us Alerts Contact Help Log in Adaptation , Learning , and Optimization over Networks , 2011 .

[23]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[24]  Julien Mairal,et al.  Structured sparsity through convex optimization , 2011, ArXiv.

[25]  Moshe Lewenstein,et al.  A faster implementation of the Goemans-Williamson clustering algorithm , 2001, SODA '01.

[26]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[27]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[28]  David P. Woodruff,et al.  Lower bounds for sparse recovery , 2010, SODA '10.

[29]  Martin J. Wainwright,et al.  A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers , 2009, NIPS.

[30]  Volkan Cevher,et al.  Sparse Signal Recovery Using Markov Random Fields , 2008, NIPS.

[31]  Aristides Gionis,et al.  Event detection in activity networks , 2014, KDD.

[32]  Lawrence Carin,et al.  Exploiting Structure in Wavelet-Based Bayesian Compressive Sensing , 2009, IEEE Transactions on Signal Processing.

[33]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.

[34]  Noah Simon,et al.  A Sparse-Group Lasso , 2013 .

[35]  David P. Williamson,et al.  A general approximation technique for constrained forest problems , 1992, SODA '92.

[36]  Lorenzo Rosasco,et al.  A Primal-Dual Algorithm for Group Sparse Regularization with Overlapping Groups , 2010, NIPS.

[37]  Piotr Indyk,et al.  Approximation-Tolerant Model-Based Compressive Sensing , 2014, SODA.

[38]  Piotr Indyk,et al.  Seismic feature extraction using steiner tree methods , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[39]  R. Maitra,et al.  Supplement to “ A k-mean-directions Algorithm for Fast Clustering of Data on the Sphere ” published in the Journal of Computational and Graphical Statistics , 2009 .

[40]  James T. Kwok,et al.  MultiLabel Classification on Tree- and DAG-Structured Hierarchies , 2011, ICML.