Revisiting the Variable Projection Method for Separable Nonlinear Least Squares Problems

Variable Projection (VarPro) is a framework to solve optimization problems efficiently by optimally eliminating a subset of the unknowns. It is in particular adapted for Separable Nonlinear Least Squares (SNLS) problems, a class of optimization problems including low-rank matrix factorization with missing data and affine bundle adjustment as instances. VarPro-based methods have received much attention over the last decade due to the experimentally observed large convergence basin for certain problem classes, where they have a clear advantage over standard methods based on Joint optimization over all unknowns. Yet no clear answers have been found in the literature as to why VarPro outperforms others and why Joint optimization, which has been successful in solving many computer vision tasks, fails on this type of problems. Also, the fact that VarPro has been mainly tested on small to medium-sized datasets has raised questions about its scalability. This paper intends to address these unsolved puzzles.

[1]  Andrew W. Fitzgibbon,et al.  Secrets of Matrix Factorization: Approximations, Numerics, Manifold Optimization and Random Restarts , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  L. Kaufman.  A variable projection method for solving separable nonlinear least squares problems , 1974 .

[3]  Richard Szeliski,et al.  Pushing the Envelope of Modern Methods for Bundle Adjustment , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Dianne P. O'Leary,et al.  Variable projection for nonlinear least squares problems , 2012, Computational Optimization and Applications.

[5]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[6]  Aleix M. Martínez,et al.  Computing Smooth Time Trajectories for Camera and Deformable Shape in Structure from Motion with Occlusion , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Tony F. Chan,et al.  Aspects of Total Variation Regularized L[sup 1] Function Approximation , 2005, SIAM J. Appl. Math..

[8]  Hiroshi Ishikawa,et al.  Exact Optimization for Markov Random Fields with Convex Priors , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Takayuki Okatani,et al.  Efficient algorithm for low-rank matrix factorization with missing components and performance comparison of latest algorithms , 2011, 2011 International Conference on Computer Vision.

[10]  Andrew W. Fitzgibbon,et al.  Projective Bundle Adjustment from Arbitrary Initialization Using the Variable Projection Method , 2016, ECCV.

[11]  M. Saunders,et al.  Solution of Sparse Indefinite Systems of Linear Equations , 1975 .

[12]  Christopher Zach,et al.  Robust Bundle Adjustment Revisited , 2014, ECCV.

[13]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[14]  David J. Kriegman,et al.  What is the set of images of an object under all possible lighting conditions? , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Pierre-Antoine Absil,et al.  RTRMC: A Riemannian trust-region method for low-rank matrix completion , 2011, NIPS.

[16]  Yair Weiss,et al.  MAP Estimation, Linear Programming and Belief Propagation with Convex Free Energies , 2007, UAI.

[17]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[18]  G. Golub,et al.  Separable nonlinear least squares: the variable projection method and its applications , 2003 .

[19]  Gene H. Golub,et al.  The differentiation of pseudo-inverses and non-linear least squares problems whose variables separate , 1972, Milestones in Matrix Computation.

[20]  Dennis Strelow,et al.  General and Nested Wiberg Minimization: L 2 and Maximum Likelihood , 2012, ECCV.

[21]  Dennis Strelow,et al.  General and nested Wiberg minimization , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Andrew W. Fitzgibbon,et al.  What Shape Are Dolphins? Building 3D Morphable Models from 2D Images , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Takayuki Okatani,et al.  On the Wiberg Algorithm for Matrix Factorization in the Presence of Missing Components , 2007, International Journal of Computer Vision.

[24]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[25]  Andrew W. Fitzgibbon,et al.  Damped Newton algorithms for matrix factorization with missing data , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[26]  James Bennett,et al.  The Netflix Prize , 2007 .

[27]  Pei Chen,et al.  Optimization Algorithms on Subspaces: Revisiting Missing Data Problem in Low-Rank Matrix , 2008, International Journal of Computer Vision.

[28]  Axel Ruhe,et al.  Algorithms for separable nonlinear least squares problems , 1980 .

[29]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).