A Comparative Study of Modern Inference Techniques for Discrete Energy Minimization Problems

Even years ago, Szeliski et al. published an influential study on energy minimization methods for Markov random fields (MRF). This study provided valuable insights in choosing the best optimization technique for certain classes of problems. While these insights remain generally useful today, the phenominal success of random field models means that the kinds of inference problems we solve have changed significantly. Specifically, the models today often include higher order interactions, flexible connectivity structures, large label-spaces of different cardinalities, or learned energy tables. To reflect these changes, we provide a modernized and enlarged study. We present an empirical comparison of 24 state-of-art techniques on a corpus of 2,300 energy minimization instances from 20 diverse computer vision applications. To ensure reproducibility, we evaluate all methods in the OpenGM2 framework and report extensive results regarding runtime and solution quality. Key insights from our study agree with the results of Szeliski et al. for the types of models they studied. However, on new and challenging types of models our findings disagree and suggest that polyhedral methods and integer programming solvers are competitive in terms of runtime and solution quality over a large range of model types.

[1]  M. R. Rao,et al.  The partition problem , 1993, Math. Program..

[2]  Carlo Tomasi,et al.  A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  Nicholas Ayache,et al.  Medical Image Analysis: Progress over Two Decades and the Challenges Ahead , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[6]  Vladimir Kolmogorov,et al.  Computing geodesics and minimal surfaces via graph cuts , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[7]  William T. Freeman,et al.  Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[8]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  R. Zabih,et al.  What energy functions can be minimized via graph cuts , 2004 .

[10]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  C. Emslie,et al.  Masculinities in Older Men: A Qualitative Study in the West of Scotland , 2004 .

[12]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[13]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[14]  Martin J. Wainwright,et al.  MAP estimation via agreement on trees: message-passing and linear programming , 2005, IEEE Transactions on Information Theory.

[15]  Alexei A. Efros,et al.  Recovering Surface Layout from an Image , 2007, International Journal of Computer Vision.

[16]  Vladimir Kolmogorov,et al.  Comparison of Energy Minimization Algorithms for Highly Connected Graphs , 2006, ECCV.

[17]  Vladimir Kolmogorov,et al.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Antonio Criminisi,et al.  TextonBoost: Joint Appearance, Shape and Context Modeling for Multi-class Object Recognition and Segmentation , 2006, ECCV.

[19]  Nikos Komodakis,et al.  Approximate Labeling via Graph Cuts Based on Linear Programming , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Vladimir Kolmogorov,et al.  Optimizing Binary MRFs via Extended Roof Duality , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Nikos Komodakis,et al.  Beyond Loose LP-Relaxations: Optimizing MRFs by Repairing Cycles , 2008, ECCV.

[22]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Christoph Schnörr,et al.  A Study of Parts-Based Object Class Detection Using Complete Graphs , 2010, International Journal of Computer Vision.

[24]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[25]  Stephen Gould,et al.  Decomposing a scene into geometric and semantically consistent regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[26]  Daphne Koller,et al.  Efficiently selecting regions for scene understanding , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  Ullrich Köthe,et al.  An Empirical Comparison of Inference Algorithms for Graphical Models with Higher Order Factors Using OpenGM , 2010, DAGM-Symposium.

[28]  Ullrich Köthe,et al.  The Lazy Flipper: MAP Inference in Higher-Order Graphical Models by Depth-limited Exhaustive Search , 2010, ArXiv.

[29]  Pushmeet Kohli,et al.  Dynamic Hybrid Algorithms for MAP Inference in Discrete MRFs , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Alexei A. Efros,et al.  Recovering Occlusion Boundaries from an Image , 2011, International Journal of Computer Vision.

[31]  Paria Mehrani,et al.  Superpixels and Supervoxels in an Energy Optimization Framework , 2010, ECCV.

[32]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Sebastian Nowozin,et al.  Higher-Order Correlation Clustering for Image Segmentation , 2011, NIPS.

[34]  Endre Boros,et al.  A graph cut algorithm for higher-order Markov Random Fields , 2011, 2011 International Conference on Computer Vision.

[35]  Jörg H. Kappes.  Inference on highly-connected discrete graphical models with applications to visual object recognition , 2011 .

[36]  Sebastian Nowozin,et al.  Variable grouping for energy minimization , 2011, CVPR 2011.

[37]  Lars Otten,et al.  Anytime AND/OR depth-first search for combinatorial optimization , 2011, AI Commun..

[38]  Sebastian Nowozin,et al.  Decision tree fields , 2011, 2011 International Conference on Computer Vision.

[39]  Gerhard Reinelt,et al.  Globally Optimal Image Partitioning by Multicuts , 2011, EMMCVPR.

[40]  Andrew C. Gallagher,et al.  Inference for order reduction in Markov random fields , 2011, CVPR 2011.

[41]  Ullrich Köthe,et al.  Probabilistic image segmentation with closedness constraints , 2011, 2011 International Conference on Computer Vision.

[42]  Eric L. Miller,et al.  Segmentation fusion for connectomics , 2011, 2011 International Conference on Computer Vision.

[43]  Christoph Schnörr,et al.  Continuous Multiclass Labeling Approaches and Algorithms , 2011, SIAM J. Imaging Sci..

[44]  Ullrich Köthe,et al.  Globally Optimal Closed-Surface Segmentation for Connectomics , 2012, ECCV.

[45]  Christoph Schnörr,et al.  A bundle approach to efficient MAP-inference by Lagrangian relaxation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Ullrich Köthe,et al.  The Lazy Flipper: Efficient Depth-Limited Exhaustive Search in Discrete Graphical Models , 2012, ECCV.

[47]  Ullrich Köthe,et al.  3D segmentation of SBFSEM images of neuropil by a graphical model over supervoxel boundaries , 2012, Medical Image Anal..

[48]  Jörg H. Kappes,et al.  OpenGM: A C++ Library for Discrete Graphical Models , 2012, ArXiv.

[49]  Daniel Cremers,et al.  A Convex Approach to Minimal Partitions , 2012, SIAM J. Imaging Sci..

[50]  Matthew Cook,et al.  Efficient automatic 3D-reconstruction of branching neurons from EM data , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Ullrich Köthe,et al.  A Discrete Chain Graph Model for 3d+t Cell Tracking with High Misdetection Robustness , 2012, ECCV.

[52]  Gerhard Reinelt,et al.  Towards Efficient and Exact MAP-Inference for Large Scale Discrete Computer Vision Problems via Combinatorial Optimization , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.