An Uncertainty-Weighted Asynchronous ADMM Method for Parallel PDE Parameter Estimation

We consider a global variable consensus ADMM algorithm for solving large-scale PDE parameter estimation problems asynchronously and in parallel. To this end, we partition the data and distribute the resulting subproblems among the available workers. Since each subproblem can be associated with different forward models and right-hand-sides, this provides ample options for tailoring the method to different applications including multi-source and multi-physics PDE parameter estimation problems. We also consider an asynchronous variant of consensus ADMM to reduce communication and latency. Our key contribution is a novel weighting scheme that empirically increases the progress made in early iterations of the consensus ADMM scheme and is attractive when using a large number of subproblems. This makes consensus ADMM competitive for solving PDE parameter estimation, which incurs immense costs per iteration. The weights in our scheme are related to the uncertainty associated with the solutions of each subproblem. We exemplarily show that the weighting scheme combined with the asynchronous implementation improves the time-to-solution for a 3D single-physics and multiphysics PDE parameter estimation problems.

[1]  William W. Hager,et al.  A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search , 2005, SIAM J. Optim..

[2]  Eric Darve,et al.  Computing entries of the inverse of a sparse matrix using the FIND algorithm , 2008, J. Comput. Phys..

[3]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[4]  Daniela Calvetti,et al.  Large-Scale Statistical Parameter Estimation in Complex Systems with an Application to Metabolic Models , 2006, Multiscale Model. Simul..

[5]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[6]  J. Sherman,et al.  Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix , 1950 .

[7]  Eldad Haber,et al.  Model Fusion and Joint Inversion , 2013, Surveys in Geophysics.

[8]  Eldad Haber,et al.  An Effective Method for Parameter Estimation with PDE Constraints with Multiple Right-Hand Sides , 2012, SIAM J. Optim..

[9]  Alexander Shapiro,et al.  The Sample Average Approximation Method for Stochastic Discrete Optimization , 2002, SIAM J. Optim..

[10]  Bart G. van Bloemen Waanders,et al.  Fast Algorithms for Bayesian Uncertainty Quantification in Large-Scale Linear Inverse Problems Based on Low-Rank Partial Hessian Approximations , 2011, SIAM J. Sci. Comput..

[11]  Marilena Mitrouli,et al.  Estimating the diagonal of matrix functions , 2018 .

[12]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[13]  Tom Goldstein,et al.  Unwrapping ADMM: Efficient Distributed Computing via Transpose Reduction , 2015, AISTATS.

[14]  Zhi-Quan Luo,et al.  Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems , 2014, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[15]  M. Heinkenschloss,et al.  Large-Scale PDE-Constrained Optimization: An Introduction , 2003 .

[16]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[17]  Zheng Xu,et al.  Adaptive Consensus ADMM for Distributed Optimization , 2017, ICML.

[18]  Nikos D. Sidiropoulos,et al.  Consensus-ADMM for General Quadratically Constrained Quadratic Programming , 2016, IEEE Transactions on Signal Processing.

[19]  Jan MacDonald,et al.  Improved Susceptibility Artifact Correction of Echo-Planar MRI using the Alternating Direction Method of Multipliers , 2018, Journal of Mathematical Imaging and Vision.

[20]  E. Weinan,et al.  Fast algorithm for extracting the diagonal of the inverse matrix with application to the electronic structure analysis of metallic systems , 2009 .

[21]  Yousef Saad,et al.  A Probing Method for Computing the Diagonal of the Matrix Inverse ∗ , 2010 .

[22]  Georgios B. Giannakis,et al.  Fast Decentralized Learning Via Hybrid Consensus Admm , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  Eran Treister,et al.  A fast marching algorithm for the factored eikonal equation , 2016, J. Comput. Phys..

[24]  J. P. Evans.  Physical properties of rocks: Fundamentals and principles of petrophysics , 1997 .

[25]  Per Christian Hansen,et al.  REGULARIZATION TOOLS: A Matlab package for analysis and solution of discrete ill-posed problems , 1994, Numerical Algorithms.

[26]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[27]  Zheng Xu,et al.  Scalable Classifiers with ADMM and Transpose Reduction , 2017, AAAI Workshops.

[28]  James T. Kwok,et al.  Asynchronous Distributed ADMM for Consensus Optimization , 2014, ICML.

[29]  Andrew M. Stuart,et al.  Inverse problems: A Bayesian perspective , 2010, Acta Numerica.

[30]  James Martin,et al.  A Computational Framework for Infinite-Dimensional Bayesian Inverse Problems Part I: The Linearized Case, with Application to Global Seismic Inversion , 2013, SIAM J. Sci. Comput..

[31]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[32]  Alan Edelman,et al.  Julia: A Fresh Approach to Numerical Computing , 2014, SIAM Rev..

[33]  Eldad Haber,et al.  Computational Methods in Geophysical Electromagnetics , 2014, Mathematics in Industry.

[34]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[35]  Timothy A. Davis,et al.  The university of Florida sparse matrix collection , 2011, TOMS.

[36]  Yousef Saad,et al.  Domain-Decomposition-Type Methods for Computing the Diagonal of a Matrix Inverse , 2011, SIAM J. Sci. Comput..

[37]  Ali Molaei,et al.  Norm-1 Regularized Consensus-Based ADMM for Imaging With a Compressive Antenna , 2017, IEEE Antennas and Wireless Propagation Letters.

[38]  W. Hager,et al.  A SURVEY OF NONLINEAR CONJUGATE GRADIENT METHODS , 2005 .

[39]  Peter K. Kitanidis,et al.  Randomized algorithms for generalized Hermitian eigenvalue problems with application to computing Karhunen–Loève expansion , 2013, Numer. Linear Algebra Appl..

[40]  Lise Getoor,et al.  A hypergraph-partitioned vertex programming approach for large-scale consensus optimization , 2013, 2013 IEEE International Conference on Big Data.

[41]  Eran Treister,et al.  jInv-a Flexible Julia Package for PDE Parameter Estimation , 2016, SIAM J. Sci. Comput..

[42]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[43]  Ali Molaei,et al.  Consensus-based imaging using ADMM for a Compressive Reflector Antenna , 2015, 2015 IEEE International Symposium on Antennas and Propagation & USNC/URSI National Radio Science Meeting.

[44]  Y. Saad,et al.  An estimator for the diagonal of a matrix , 2007 .