Guarantees for Greedy Maximization of Non-submodular Functions with Applications

We investigate the performance of the standard GREEDY algorithm for cardinality constrained maximization of non-submodular nondecreasing set functions. While there are strong theoretical guarantees on the performance of GREEDY for maximizing submodular functions, there are few guarantees for non-submodular ones. However, GREEDY enjoys strong empirical performance for many important non-submodular functions, e.g., the Bayesian A-optimality objective in experimental design. We prove theoretical guarantees supporting the empirical performance. Our guarantees are characterized by a combination of the (generalized) curvature α and the sub-modularity ratio γ. In particular, we prove that GREEDY enjoys a tight approximation guarantee of 1/α (1 - e-γα) for cardinality constrained maximization. In addition, we bound the submod-ularity ratio and curvature for several important real-world objectives, including the Bayesian A-optimality objective, the determinantal function of a square submatrix and certain linear programs with combinatorial constraints. We experimentally validate our theoretical findings for both synthetic and real-world applications.

[1]  Jan Vondrák,et al.  Optimal approximation for submodular and supermodular optimization with bounded curvature , 2013, SODA.

[2]  Panos M. Pardalos,et al.  Analysis of greedy approximations with nonsubmodular potential functions , 2008, SODA '08.

[3]  Francis R. Bach,et al.  Learning with Submodular Functions: A Convex Optimization Perspective , 2011, Found. Trends Mach. Learn..

[4]  Andreas Krause,et al.  Guaranteed Non-convex Optimization: Submodular Maximization over Continuous Domains , 2016, AISTATS.

[5]  Uriel Feige,et al.  Welfare maximization and the supermodular degree , 2013, ITCS '13.

[6]  Abhimanyu Das,et al.  Algorithms for subset selection in linear regression , 2008, STOC.

[7]  Andreas Krause,et al.  Near-optimal Nonmyopic Value of Information in Graphical Models , 2005, UAI.

[8]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[9]  Pushmeet Kohli,et al.  Tractability: Practical Approaches to Hard Problems , 2013 .

[10]  Rishabh K. Iyer,et al.  Algorithms for Approximate Minimization of the Difference Between Submodular Functions, with Applications , 2012, UAI.

[11]  Rishabh K. Iyer,et al.  Curvature and Optimal Algorithms for Learning and Minimizing Submodular Functions , 2013, NIPS.

[12]  Alexandros G. Dimakis,et al.  Restricted Strong Convexity Implies Weak Submodularity , 2016, The Annals of Statistics.

[13]  Abhimanyu Das,et al.  Submodular meets Spectral: Greedy Algorithms for Subset Selection, Sparse Approximation and Dictionary Selection , 2011, ICML.

[14]  Costas J. Spanos,et al.  Causal meets Submodular: Subset Selection with Directed Information , 2016, NIPS.

[15]  Rishabh K. Iyer,et al.  Algorithms for Optimizing the Ratio of Submodular Functions , 2016, ICML.

[16]  Joseph Naor,et al.  Submodular Maximization with Cardinality Constraints , 2014, SODA.

[17]  Andreas Krause,et al.  Submodular Function Maximization , 2014, Tractability.

[18]  Prateek Jain,et al.  On Iterative Hard Thresholding Methods for High-dimensional M-Estimation , 2014, NIPS.

[19]  Aditya Bhaskara,et al.  Greedy Column Subset Selection: New Bounds and Distributed Algorithms , 2016, ICML.

[20]  Ken-ichi Kawarabayashi,et al.  Optimal Budget Allocation: Theoretical Guarantee and Efficient Algorithm , 2014, ICML.

[21]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[22]  Andreas Krause,et al.  Submodular Dictionary Selection for Sparse Representation , 2010, ICML.

[23]  Gérard Cornuéjols,et al.  Submodular set functions, matroids and the greedy algorithm: Tight worst-case bounds and some generalizations of the Rado-Edmonds theorem , 1984, Discret. Appl. Math..

[24]  Rishabh K. Iyer,et al.  On Approximate Non-submodular Minimization via Tree-Structured Supermodularity , 2015, AISTATS.

[25]  Jeff A. Bilmes,et al.  Submodularity beyond submodular energies: Coupling edges in graph cuts , 2011, CVPR 2011.

[26]  K. Chaloner,et al.  Bayesian Experimental Design: A Review , 1995 .

[27]  Yaron Singer,et al.  Maximization of Approximately Submodular Functions , 2016, NIPS.

[28]  Jeff A. Bilmes,et al.  A Submodular-supermodular Procedure with Applications to Discriminative Structure Learning , 2005, UAI.

[29]  M. L. Fisher,et al.  An analysis of approximations for maximizing submodular set functions—I , 1978, Math. Program..

[30]  J. Vondrák.  Submodularity and curvature : the optimal algorithm , 2008 .

[31]  Andreas Krause,et al.  Near-Optimal Sensor Placements in Gaussian Processes: Theory, Efficient Algorithms and Empirical Studies , 2008, J. Mach. Learn. Res..

[32]  Jan Vondr.  Submodularity and Curvature: The Optimal Algorithm , 2010 .

[33]  Neil D. Lawrence,et al.  Fast Sparse Gaussian Process Methods: The Informative Vector Machine , 2002, NIPS.

[34]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[35]  Allan Borodin,et al.  Weakly Submodular Functions , 2014, ArXiv.

[36]  Jan Vondrák,et al.  Optimal approximation for the submodular welfare problem in the value oracle model , 2008, STOC.

[37]  Yuichi Yoshida,et al.  Maximizing a Monotone Submodular Function with a Bounded Curvature under a Knapsack Constraint , 2016, SIAM J. Discret. Math..

[38]  Maxim Sviridenko,et al.  A note on maximizing a submodular set function subject to a knapsack constraint , 2004, Oper. Res. Lett..

[39]  John N. Tsitsiklis,et al.  Introduction to linear optimization , 1997, Athena scientific optimization and computation series.