Probabilistic Permutation Synchronization Using the Riemannian Structure of the Birkhoff Polytope

We present an entirely new geometric and probabilistic approach to synchronization of correspondences across multiple sets of objects or images. In particular, we present two algorithms: (1) Birkhoff-Riemannian L-BFGS for optimizing the relaxed version of the combinatorially intractable cycle consistency loss in a principled manner, (2) Birkhoff-Riemannian Langevin Monte Carlo for generating samples on the Birkhoff Polytope and estimating the confidence of the found solutions. To this end, we first introduce the very recently developed Riemannian geometry of the Birkhoff Polytope. Next, we introduce a new probabilistic synchronization model in the form of a Markov Random Field (MRF). Finally, based on the first order retraction operators, we formulate our problem as simulating a stochastic differential equation and devise new integrators. We show on both synthetic and real datasets that we achieve high quality multi-graph matching results with faster convergence and reliable confidence/uncertainty estimates.

[1]  Nan Hu,et al.  Distributable Consistent Multi-object Matching , 2016, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  Slobodan Ilic,et al.  PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors , 2018, ECCV.

[3]  Vikas Singh,et al.  Permutation Diffusion Maps (PDM) with Application to the Image Association Problem in Computer Vision , 2014, NIPS.

[4]  Martial Hebert,et al.  Fully automatic registration of multiple 3D data sets , 2003, Image Vis. Comput..

[5]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[6]  Johan Thunberg,et al.  Synchronisation of Partial Multi-Matchings via Non-negative Factorisations , 2018, Pattern Recognit..

[7]  Timothy Bretl,et al.  Improved Structure from Motion Using Fiducial Marker Matching , 2018, ECCV.

[8]  Guillermo Sapiro,et al.  Graph Matching: Relax at Your Own Risk , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  M. Girolami,et al.  Geodesic Monte Carlo on Embedded Manifolds , 2013, Scandinavian journal of statistics, theory and applications.

[10]  Wen Huang,et al.  A Broyden Class of Quasi-Newton Methods for Riemannian Optimization , 2015, SIAM J. Optim..

[11]  Andrea Fusiello,et al.  Spectral Synchronization of Multiple Views in SE(3) , 2016, SIAM J. Imaging Sci..

[12]  Richard Sinkhorn,et al.  Concerning nonnegative matrices and doubly stochastic matrices , 1967 .

[13]  Steven Thomas Smith,et al.  Optimization Techniques on Riemannian Manifolds , 2014, ArXiv.

[14]  J. Laurie Snell,et al.  Markov Random Fields and Their Applications , 1980 .

[15]  Andrea Fusiello,et al.  Practical and Efficient Multi-view Matching , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  João Paulo Costeira,et al.  A Global Solution to Sparse Correspondence Problems , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[18]  Jérémie Jakubowicz,et al.  Kantorovich Distances between Rankings with Applications to Rank Aggregation , 2010, ECML/PKDD.

[19]  Leonidas J. Guibas,et al.  Near-Optimal Joint Object Matching via Convex Relaxation , 2014, ICML.

[20]  Slobodan Ilic,et al.  PPFNet: Global Context Aware Local Features for Robust 3D Point Matching , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Slobodan Ilic,et al.  Bayesian Pose Graph Optimization via Bingham Distributions and Tempered Geodesic MCMC , 2018, NeurIPS.

[22]  Tianqi Chen,et al.  A Complete Recipe for Stochastic Gradient MCMC , 2015, NIPS.

[23]  Michel X. Goemans,et al.  Smallest compact formulation for the permutahedron , 2015, Math. Program..

[24]  Yoel Shkolnisky,et al.  Three-Dimensional Structure Determination from Common Lines in Cryo-EM by Eigenvectors and Semidefinite Programming , 2011, SIAM J. Imaging Sci..

[25]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[26]  Torsten Sattler,et al.  Image Retrieval for Image-Based Localization Revisited , 2012, BMVC.

[27]  Yee Whye Teh,et al.  Stochastic Gradient Riemannian Langevin Dynamics on the Probability Simplex , 2013, NIPS.

[28]  J. Munkres.  ALGORITHMS FOR THE ASSIGNMENT AND TRANSIORTATION tROBLEMS* , 1957 .

[29]  Ieee Xplore,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Information for Authors , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Roland Badeau,et al.  Stochastic Quasi-Newton Langevin Monte Carlo , 2016, ICML.

[31]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Amit Singer,et al.  Global Registration of Multiple Point Clouds Using Semidefinite Programming , 2013, SIAM J. Optim..

[33]  G. Hurlbert,et al.  A SHORT PROOF OF THE BIRKHOFF-VON NEUMANN THEOREM , 2012 .

[34]  Martha Larson,et al.  Pairwise geometric matching for large-scale object retrieval , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Wen Huang,et al.  ROPTLIB , 2018, ACM Trans. Math. Softw..

[36]  Vince D. Calhoun,et al.  Directional Statistics on Permutations , 2010, AISTATS.

[37]  Hongyuan Zha,et al.  Multi-Graph Matching via Affinity Optimization with Graduated Consistency Regularization , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Yifan Sun,et al.  Joint Map and Symmetry Synchronization , 2018, ECCV.

[39]  Xinhua Zhang,et al.  Efficient and Consistent Adversarial Bipartite Matching , 2018, ICML.

[40]  A. Singer.  Angular Synchronization by Eigenvectors and Semidefinite Programming. , 2009, Applied and computational harmonic analysis.

[41]  Benedikt Wirth,et al.  Optimization Methods on Riemannian Manifolds and Their Application to Shape Space , 2012, SIAM J. Optim..

[42]  Amit Singer,et al.  Exact and Stable Recovery of Rotations for Robust Synchronization , 2012, ArXiv.

[43]  Bruno Iannazzo,et al.  The Riemannian Barzilai–Borwein method with nonmonotone line search and the matrix geometric mean computation , 2018 .

[44]  M. Girolami,et al.  Riemann manifold Langevin and Hamiltonian Monte Carlo methods , 2011, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[45]  Matthias Nießner,et al.  BundleFusion , 2016, TOGS.

[46]  Lawrence Carin,et al.  On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators , 2015, NIPS.

[47]  Johan Thunberg,et al.  A solution for multi-alignment by transformation synchronisation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Cristian Sminchisescu,et al.  Deep Learning of Graph Matching , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  Umut Simsekli,et al.  Fractional Langevin Monte Carlo: Exploring Levy Driven Stochastic Differential Equations for Markov Chain Monte Carlo , 2017, ICML.

[50]  Xiaowei Zhou,et al.  Multi-image Semantic Matching by Mining Consistent Features , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[51]  Hongdong Li,et al.  The 3D-3D Registration Problem Revisited , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[52]  Daniel Cremers,et al.  Consistent Partial Matching of Shape Collections via Sparse Modeling , 2017, Comput. Graph. Forum.

[53]  Note on the geodesic Monte Carlo , 2018, 1805.05289.

[54]  Eric Moulines,et al.  Stochastic Gradient Richardson-Romberg Markov Chain Monte Carlo , 2016, NIPS.

[55]  Babak Hassibi,et al.  Manifold Optimization Over the Set of Doubly Stochastic Matrices: A Second-Order Geometry , 2018, IEEE Transactions on Signal Processing.

[56]  Juan D. Tardós,et al.  ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras , 2016, IEEE Transactions on Robotics.

[57]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[58]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[59]  Siam Rfview,et al.  CONVERGENCE CONDITIONS FOR ASCENT METHODS , 2016 .

[60]  P. Diaconis,et al.  Sampling From A Manifold , 2012, 1206.6913.

[61]  Yang Song,et al.  Stochastic Gradient Geodesic MCMC Methods , 2016, NIPS.

[62]  Da Tang,et al.  Initialization and Coordinate Optimization for Multi-way Matching , 2016, AISTATS.

[63]  Scott W. Linderman,et al.  Reparameterizing the Birkhoff Polytope for Variational Permutation Inference , 2017, AISTATS.

[64]  Leonidas J. Guibas,et al.  An Optimization Approach to Improving Collections of Shape Maps , 2011, Comput. Graph. Forum.

[65]  Birdal Tolga,et al.  Online inspection of 3D parts via a locally overlapping camera network , 2016 .

[66]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[67]  L. Armijo.  Minimization of functions having Lipschitz continuous first partial derivatives. , 1966 .

[68]  Leonidas J. Guibas,et al.  Consistent Shape Maps via Semidefinite Programming , 2013, SGP '13.

[69]  M. Girolami,et al.  Langevin diffusions and the Metropolis-adjusted Langevin algorithm , 2013, 1309.2983.

[70]  Slobodan Ilic,et al.  CAD Priors for Accurate and Flexible Instance Reconstruction , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[71]  Pierre-Antoine Absil,et al.  Riemannian BFGS Algorithm with Applications , 2010 .

[72]  Andrea Fusiello,et al.  Synchronization in the Symmetric Inverse Semigroup , 2017, ICIAP.

[73]  Jean Ponce,et al.  Learning Graphs to Match , 2013, 2013 IEEE International Conference on Computer Vision.

[74]  Ali Taylan Cemgil,et al.  Asynchronous Stochastic Quasi-Newton MCMC for Non-Convex Optimization , 2018, ICML.

[75]  Fanny Dufossé,et al.  Notes on Birkhoff-von Neumann decomposition of doubly stochastic matrices , 2016 .

[76]  Gui-Song Xia,et al.  Globally consistent correspondence of multiple feature sets using proximal Gauss-Seidel relaxation , 2016, Pattern Recognit..

[77]  Alexander M. Bronstein,et al.  Numerical Geometry of Non-Rigid Shapes , 2009, Monographs in Computer Science.

[78]  Jan-Michael Frahm,et al.  Structure-from-Motion Revisited , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Nassir Navab,et al.  A Stereo Vision Approach for Cooperative Robotic Movement Therapy , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[80]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[81]  Michael I. Jordan,et al.  Averaging Stochastic Gradient Descent on Riemannian Manifolds , 2018, COLT.

[82]  Venu Madhav Govindu.  Lie-algebraic averaging for globally consistent motion estimation , 2004, CVPR 2004.

[83]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[84]  Daniel Cremers,et al.  A game-theoretical approach for joint matching of multiple feature throughout unordered images , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[85]  Florence d'Alché-Buc,et al.  A Structured Prediction Approach for Label Ranking , 2018, NeurIPS.

[86]  René Vidal,et al.  Distributed 3-D Localization of Camera Sensor Networks From 2-D Image Measurements , 2014, IEEE Transactions on Automatic Control.

[87]  Hongyuan Zha,et al.  A Short Survey of Recent Advances in Graph Matching , 2016, ICMR.

[88]  Xiaowei Zhou,et al.  Multi-image Matching via Fast Alternating Minimization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[89]  Vikas Singh,et al.  Solving the multi-way matching problem by permutation synchronization , 2013, NIPS.

[90]  Wen Huang,et al.  A Riemannian Limited-memory BFGS Algorithm for Computing the Matrix Geometric Mean , 2016, ICCS.

[91]  Andrea Torsello,et al.  Synchronization Over the Birkhoff Polytope for Multi-graph Matching , 2017, GbRPR.

[92]  W. Huang,et al.  Line Search Algorithms for Locally Lipschitz Functions on Riemannian Manifolds , 2018, SIAM J. Optim..

[93]  Kuk-Jin Yoon,et al.  Consistent multiple graph matching with multi-layer random walks synchronization , 2017, Pattern Recognit. Lett..

[94]  Johan Thunberg,et al.  Distributed methods for synchronization of orthogonal matrices over graphs , 2017, Autom..

[95]  Yao Lu,et al.  A fast projected fixed-point algorithm for large graph matching , 2012, Pattern Recognit..

[96]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[97]  M. Zaslavskiy,et al.  A Path Following Algorithm for the Graph Matching Problem , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.