Prismatic Algorithm for Discrete D.C. Programming Problem

In this paper, we propose the first exact algorithm for minimizing the difference of two submodular functions (D.S.), i.e., the discrete version of the D.C. programming problem. The developed algorithm is a branch-and-bound-based algorithm which responds to the structure of this problem through the relationship between submodularity and convexity. The D.S. programming problem covers a broad range of applications in machine learning. In fact, this generalizes any set-function optimization. We empirically investigate the performance of our algorithm, and illustrate the difference between exact and approximate solutions respectively obtained by the proposed and existing algorithms in feature selection and discriminative structure learning.

[1]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[2]  R. Horst,et al.  Global Optimization: Deterministic Approaches , 1992 .

[3]  Franz Pernkopf,et al.  Discriminative versus generative parameter and structure learning of Bayesian network classifiers , 2005, ICML.

[4]  Jeff A. Bilmes,et al.  A Submodular-supermodular Procedure with Applications to Discriminative Structure Learning , 2005, UAI.

[5]  Andreas Krause,et al.  Submodular Dictionary Selection for Sparse Representation , 2010, ICML.

[6]  Kazuo Murota,et al.  Discrete convex analysis , 1998, Math. Program..

[7]  Jeff A. Bilmes,et al.  Submodularity Cuts and Applications , 2009, NIPS.

[8]  László Lovász,et al.  Submodular functions and convexity , 1982, ISMP.

[9]  H. B. McMahan,et al.  Robust Submodular Observation Selection , 2008 .

[10]  藤重 悟.  Submodular functions and optimization , 1991 .

[11]  Philip S. Yu,et al.  Near-optimal Supervised Feature Selection among Frequent Subgraphs , 2009, SDM.

[12]  Jeff A. Bilmes,et al.  Dynamic Bayesian Multinets , 2000, UAI.

[13]  Jeff A. Bilmes,et al.  PAC-learning Bounded Tree-width Graphical Models , 2004, UAI.

[14]  Francis R. Bach,et al.  Structured sparsity-inducing norms through submodular functions , 2010, NIPS.

[15]  Maurice Queyranne,et al.  Minimizing symmetric submodular functions , 1998, Math. Program..

[16]  茨木 俊秀,et al.  Enumerative approaches to combinatorial optimization , 1987 .

[17]  Reiner Horst,et al.  On solving a D.C. programming problem by a sequence of linear programs , 1991, J. Glob. Optim..

[18]  S. Fujishige,et al.  The Minimum-Norm-Point Algorithm Applied to Submodular Function Minimization and Linear Programming , 2006 .

[19]  Elad Hazan,et al.  Beyond Convexity: Online Submodular Minimization , 2009, NIPS.

[20]  Satoru Iwata,et al.  Minimum Average Cost Clustering , 2010, NIPS.

[21]  Abhimanyu Das,et al.  Algorithms for subset selection in linear regression , 2008, STOC.

[22]  T. Ibaraki.  Enumerative approaches to combinatorial optimization - part I , 1988 .

[23]  Alexander Shekhovtsov,et al.  SUPERMODULAR DECOMPOSITION OF STRUCTURAL LABELING PROBLEM , 2005 .

[24]  Andrew Blake,et al.  Cosegmentation of Image Pairs by Histogram Matching - Incorporating a Global Constraint into MRFs , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[25]  Rong Jin,et al.  Batch mode active learning and its application to medical image classification , 2006, ICML.

[27]  Philip H. S. Torr,et al.  LP-relaxation of Binarized Energy Minimization ( Version 1 . 50 ) , 2008 .

[28]  Jack Edmonds,et al.  Submodular Functions, Matroids, and Certain Polyhedra , 2001, Combinatorial Optimization.

[29]  Alexander J. Smola,et al.  Learning Graph Matching , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.