A Comparative Study of Modern Inference Techniques for Structured Discrete Energy Minimization Problems

Szeliski et al. published an influential study in 2006 on energy minimization methods for Markov random fields. This study provided valuable insights in choosing the best optimization technique for certain classes of problems. While these insights remain generally useful today, the phenomenal success of random field models means that the kinds of inference problems that have to be solved changed significantly. Specifically, the models today often include higher order interactions, flexible connectivity structures, large label-spaces of different cardinalities, or learned energy tables. To reflect these changes, we provide a modernized and enlarged study. We present an empirical comparison of more than 27 state-of-the-art optimization techniques on a corpus of 2453 energy minimization instances from diverse applications in computer vision. To ensure reproducibility, we evaluate all methods in the OpenGM 2 framework and report extensive results regarding runtime and solution quality. Key insights from our study agree with the results of Szeliski et al. for the types of models they studied. However, on new and challenging types of models our findings disagree and suggest that polyhedral methods and integer programming solvers are competitive in terms of runtime and solution quality over a large range of model types.

[1]  Eric P. Xing,et al.  An Augmented Lagrangian Approach to Constrained MAP Inference , 2011, ICML.

[2]  Ullrich Köthe,et al.  Globally Optimal Closed-Surface Segmentation for Connectomics , 2012, ECCV.

[3]  Pushmeet Kohli,et al.  Robust Higher Order Potentials for Enforcing Label Consistency , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Anton Osokin,et al.  Fast Approximate Energy Minimization with Label Costs , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[5]  Tomás Werner,et al.  A Linear Programming Approach to Max-Sum Problem: A Review , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Christoph Schnörr,et al.  A Study of Parts-Based Object Class Detection Using Complete Graphs , 2010, International Journal of Computer Vision.

[7]  Nikos Komodakis,et al.  MRF Energy Minimization and Beyond via Dual Decomposition , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  N. Metropolis,et al.  An Efficient Heuristic Procedure for Partitioning Graphs , 2017 .

[9]  J. Besag.  On the Statistical Analysis of Dirty Pictures , 1986 .

[10]  Monique Guignard-Spielberg,et al.  Lagrangean decomposition: A model yielding stronger lagrangean bounds , 1987, Math. Program..

[11]  George Papandreou,et al.  Perturb-and-MAP random fields: Using discrete optimization to learn and sample from energy models , 2011, 2011 International Conference on Computer Vision.

[12]  Ulrik Brandes,et al.  On Modularity Clustering , 2008, IEEE Transactions on Knowledge and Data Engineering.

[13]  Bogdan Savchynskyy,et al.  Getting Feasible Variable Estimates from Infeasible Ones: MRF Local Polytope Study , 2012, 2013 IEEE International Conference on Computer Vision Workshops.

[14]  Sebastian Nowozin,et al.  A Comparative Study of Modern Inference Techniques for Discrete Energy Minimization Problems , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Vladimir Kolmogorov,et al.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Ullrich Köthe,et al.  A Discrete Chain Graph Model for 3d+t Cell Tracking with High Misdetection Robustness , 2012, ECCV.

[17]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[18]  Joseph Naor,et al.  A Linear Programming Formulation and Approximation Algorithms for the Metric Labeling Problem , 2005, SIAM J. Discret. Math..

[19]  Thomas Stützle,et al.  Efficient Stochastic Local Search for MPE Solving , 2005, IJCAI.

[20]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[21]  Ullrich Köthe,et al.  Probabilistic image segmentation with closedness constraints , 2011, 2011 International Conference on Computer Vision.

[22]  Ullrich Köthe,et al.  An Empirical Comparison of Inference Algorithms for Graphical Models with Higher Order Factors Using OpenGM , 2010, DAGM-Symposium.

[23]  大西 仁,et al.  Pearl, J. (1988, second printing 1991). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan-Kaufmann. , 1994 .

[24]  Gerhard Reinelt,et al.  Towards Efficient and Exact MAP-Inference for Large Scale Discrete Computer Vision Problems via Combinatorial Optimization , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Pushmeet Kohli,et al.  Making the right moves: Guiding alpha-expansion using local primal-dual gaps , 2011, CVPR 2011.

[26]  Thorsten Koch,et al.  Branching rules revisited , 2005, Oper. Res. Lett..

[27]  Alan C. Evans,et al.  BrainWeb: Online Interface to a 3D MRI Simulated Brain Database , 1997 .

[28]  Vladimir Kolmogorov,et al.  Dynamic Tree Block Coordinate Ascent , 2011, ICML.

[29]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[30]  Gerhard Reinelt,et al.  Globally Optimal Image Partitioning by Multicuts , 2011, EMMCVPR.

[31]  Daniel Cremers,et al.  A Survey and Comparison of Discrete and Continuous Multi-label Optimization Approaches for the Potts Model , 2013, International Journal of Computer Vision.

[32]  Ullrich Köthe,et al.  The Lazy Flipper: Efficient Depth-Limited Exhaustive Search in Discrete Graphical Models , 2012, ECCV.

[33]  Pushmeet Kohli,et al.  Reduce, reuse & recycle: Efficiently solving multi-label MRFs , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Christoph Schnörr,et al.  MAP-Inference on Large Scale Higher-Order Discrete Graphical Models by Fusion Moves , 2014, ECCV Workshops.

[35]  Andrew Blake,et al.  Digital tapestry [automatic image synthesis] , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[36]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[37]  Vladimir Kolmogorov,et al.  Computing geodesics and minimal surfaces via graph cuts , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[38]  Brian W. Kernighan,et al.  An efficient heuristic procedure for partitioning graphs , 1970, Bell Syst. Tech. J..

[39]  Michael I. Jordan.  Graphical Models , 1998 .

[40]  Andrew C. Gallagher,et al.  Inference for order reduction in Markov random fields , 2011, CVPR 2011.

[41]  David Sontag,et al.  Efficiently Searching for Frustrated Cycles in MAP Inference , 2012, UAI.

[42]  Sebastian Nowozin,et al.  Decision tree fields , 2011, 2011 International Conference on Computer Vision.

[43]  Christoph Schnörr,et al.  Continuous Multiclass Labeling Approaches and Algorithms , 2011, SIAM J. Imaging Sci..

[44]  Yuval Rabani,et al.  An improved approximation algorithm for multiway cut , 1998, STOC '98.

[45]  Gerhard Reinelt,et al.  Higher-order segmentation via multicuts , 2013, Comput. Vis. Image Underst..

[46]  Martin J. Wainwright,et al.  MAP estimation via agreement on trees: message-passing and linear programming , 2005, IEEE Transactions on Information Theory.

[47]  Lena Gorelick,et al.  Submodularization for Binary Pairwise Energies , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[48]  Ullrich Köthe,et al.  3D segmentation of SBFSEM images of neuropil by a graphical model over supervoxel boundaries , 2012, Medical Image Anal..

[49]  Jörg H. Kappes,et al.  OpenGM: A C++ Library for Discrete Graphical Models , 2012, ArXiv.

[50]  Nikos Komodakis,et al.  Approximate Labeling via Graph Cuts Based on Linear Programming , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Anand D. Sarwate,et al.  On Measure Concentration of Random Maximum A-Posteriori Perturbations , 2013, ICML.

[52]  Sebastian Nowozin,et al.  Higher-Order Correlation Clustering for Image Segmentation , 2011, NIPS.

[53]  Michael Jünger,et al.  Lifting and separation procedures for the cut polytope , 2014, Math. Program..

[54]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Daniel Prusa,et al.  Universality of the Local Marginal Polytope , 2013, CVPR.

[56]  Lars Otten,et al.  Anytime AND/OR depth-first search for combinatorial optimization , 2011, AI Commun..

[57]  Tomás Werner,et al.  Universality of the Local Marginal Polytope , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Vladimir Kolmogorov,et al.  Optimizing Binary MRFs via Extended Roof Duality , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[59]  Sebastian Nowozin,et al.  Variable grouping for energy minimization , 2011, CVPR 2011.

[60]  Dhruv Batra,et al.  MaxFlow Revisited: An Empirical Comparison of Maxflow Algorithms for Dense Vision Problems , 2012, BMVC.

[61]  Endre Boros,et al.  A graph cut algorithm for higher-order Markov Random Fields , 2011, 2011 International Conference on Computer Vision.

[62]  Yair Weiss,et al.  Minimizing and Learning Energy Functions for Side-Chain Prediction , 2007, RECOMB.

[63]  Vladimir Kolmogorov,et al.  Submodular decomposition framework for inference in associative Markov networks with global constraints , 2011, CVPR 2011.

[64]  Christoph Schnörr,et al.  A bundle approach to efficient MAP-inference by Lagrangian relaxation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[65]  Vladimir Kolmogorov,et al.  Comparison of Energy Minimization Algorithms for Highly Connected Graphs , 2006, ECCV.

[66]  Éva Tardos,et al.  Approximation algorithms for classification problems with pairwise relationships: metric labeling and Markov random fields , 2002, JACM.

[67]  Nikos Komodakis,et al.  Beyond Loose LP-Relaxations: Optimizing MRFs by Repairing Cycles , 2008, ECCV.

[68]  Alexei A. Efros,et al.  Recovering Occlusion Boundaries from an Image , 2011, International Journal of Computer Vision.

[69]  Ivan Kovtun,et al.  Partial Optimal Labeling Search for a NP-Hard Subclass of (max, +) Problems , 2003, DAGM-Symposium.

[70]  Stephen Gould,et al.  Decomposing a scene into geometric and semantically consistent regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[71]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  Tommi S. Jaakkola,et al.  Fixing Max-Product: Convergent Message Passing Algorithms for MAP LP-Relaxations , 2007, NIPS.

[73]  Andrew Blake,et al.  Fusion Moves for Markov Random Field Optimization , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Sebastian Nowozin,et al.  Structured Learning and Prediction in Computer Vision , 2011, Found. Trends Comput. Graph. Vis..

[75]  Christoph Schnörr,et al.  Global MAP-Optimality by Shrinking the Combinatorial Search Area with Convex Relaxation , 2013, NIPS.

[76]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[77]  Andrew W. Fitzgibbon,et al.  Global stereo reconstruction under second order smoothness priors , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[78]  Pushmeet Kohli,et al.  Dynamic Hybrid Algorithms for MAP Inference in Discrete MRFs , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  Christoph Schnörr,et al.  Efficient MRF Energy Minimization via Adaptive Diminishing Smoothing , 2012, UAI.

[80]  Nir Friedman,et al.  Towards an Integrated Protein-Protein Interaction Network: A Relational Markov Network Approach , 2006, J. Comput. Biol..