Optimal DR-Submodular Maximization and Applications to Provable Mean Field Inference

Mean field inference in probabilistic models is generally a highly nonconvex problem. Existing optimization methods, e.g., coordinate ascent algorithms, can only generate local optima. In this work we propose provable mean filed methods for probabilistic log-submodular models and its posterior agreement (PA) with strong approximation guarantees. The main algorithmic technique is a new Double Greedy scheme, termed DR-DoubleGreedy, for continuous DR-submodular maximization with box-constraints. It is a one-pass algorithm with linear time complexity, reaching the optimal 1/2 approximation ratio, which may be of independent interest. We validate the superior performance of our algorithms against baseline algorithms on both synthetic and real-world datasets.

[1]  Joseph Naor,et al.  A Tight Linear Time (1/2)-Approximation for Unconstrained Submodular Maximization , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.

[2]  Ben Taskar,et al.  Expectation-Maximization for Learning Determinantal Point Processes , 2014, NIPS.

[3]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[4]  Yuichi Yoshida,et al.  Non-Monotone DR-Submodular Function Maximization , 2016, AAAI.

[5]  Andreas Krause,et al.  Guarantees for Greedy Maximization of Non-submodular Functions with Applications , 2017, ICML.

[6]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[7]  Andreas Krause,et al.  Submodular Function Maximization , 2014, Tractability.

[8]  Andreas Krause,et al.  From MAP to Marginals: Variational Inference in Bayesian Submodular Models , 2014, NIPS.

[9]  Joachim M. Buhmann,et al.  Information-theoretic analysis of MaxCut algorithms , 2016, 2016 Information Theory and Applications Workshop (ITA).

[10]  Andreas Krause,et al.  Near-optimal Nonmyopic Value of Information in Graphical Models , 2005, UAI.

[11]  Amin Karbasi,et al.  Online Continuous Submodular Maximization , 2018, AISTATS.

[12]  Ben Taskar,et al.  Near-Optimal MAP Inference for Determinantal Point Processes , 2012, NIPS.

[13]  Jan Vondrák,et al.  On Multiplicative Weight Updates for Concave and Submodular Function Maximization , 2015, ITCS.

[14]  Stefan Bauer,et al.  Model Selection for Gaussian Process Regression , 2017, GCPR.

[15]  Britta Peis,et al.  Submodular Function Maximization on the Bounded Integer Lattice , 2015, WAOA.

[16]  Huy L. Nguyen,et al.  A Reduction for Optimizing Lattice Submodular Functions with Diminishing Returns , 2016, ArXiv.

[17]  Vahab S. Mirrokni,et al.  Maximizing Non-Monotone Submodular Functions , 2011, 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07).

[18]  Joachim M. Buhmann,et al.  How informative are Minimum Spanning Tree algorithms? , 2014, 2014 IEEE International Symposium on Information Theory.

[19]  Stefanie Jegelka,et al.  Robust Budget Allocation Via Continuous Submodular Functions , 2017, Applied Mathematics & Optimization.

[20]  Hui Lin,et al.  Optimal Selection of Limited Vocabulary Speech Corpora , 2011, INTERSPEECH.

[21]  Jan Vondrák,et al.  Maximizing a Submodular Set Function Subject to a Matroid Constraint (Extended Abstract) , 2007, IPCO.

[22]  Hui Lin,et al.  A Class of Submodular Functions for Document Summarization , 2011, ACL.

[23]  Bryan Wilder,et al.  Risk-Sensitive Submodular Optimization , 2018, AAAI.

[24]  Tim Roughgarden,et al.  Optimal Algorithms for Continuous Non-monotone Submodular and DR-Submodular Maximization , 2018, NeurIPS.

[25]  Andreas Krause,et al.  Variational Inference in Mixed Probabilistic Submodular Models , 2016, NIPS.

[26]  Andreas Krause,et al.  Continuous DR-submodular Maximization: Structure and Algorithms , 2017, NIPS 2017.

[27]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[28]  Andreas Krause,et al.  Submodular Dictionary Selection for Sparse Representation , 2010, ICML.

[29]  Joachim M. Buhmann.  Information theoretic model validation for clustering , 2010, 2010 IEEE International Symposium on Information Theory.

[30]  László Lovász,et al.  Submodular functions and convexity , 1982, ISMP.

[31]  Rishabh K. Iyer,et al.  Submodular Point Processes with Applications to Machine learning , 2015, AISTATS.

[32]  Amin Karbasi,et al.  Decentralized Submodular Maximization: Bridging Discrete and Continuous Settings , 2018, ICML.

[33]  J. Vondrák,et al.  Submodular Function Maximization via the Multilinear Relaxation and Contention Resolution Schemes , 2014 .

[34]  Andreas Krause,et al.  Learning Probabilistic Submodular Diversity Models Via Noise Contrastive Estimation , 2016, AISTATS.

[35]  Joachim M. Buhmann,et al.  Posterior agreement for large parameter-rich optimization problems , 2018, Theor. Comput. Sci..

[36]  Ben Taskar,et al.  Determinantal Point Processes for Machine Learning , 2012, Found. Trends Mach. Learn..

[37]  Jan Vondrák,et al.  Optimal approximation for the submodular welfare problem in the value oracle model , 2008, STOC.

[38]  Andreas Krause,et al.  Guaranteed Non-convex Optimization: Submodular Maximization over Continuous Domains , 2016, AISTATS.

[39]  Andreas Krause,et al.  Scalable Variational Inference in Log-supermodular Models , 2015, ICML.

[40]  Francis Bach,et al.  Submodular functions: from discrete to continuous domains , 2015, Mathematical Programming.