The ADMM-PINNs Algorithmic Framework for Nonsmooth PDE-Constrained Optimization: A Deep Learning Approach

We study the combination of the alternating direction method of multipliers (ADMM) with physics-informed neural networks (PINNs) for a general class of nonsmooth partial differential equation (PDE)-constrained optimization problems, where additional regularization can be employed for constraints on the control or design variables. The resulting ADMM-PINNs algorithmic framework substantially enlarges the applicable range of PINNs to nonsmooth cases of PDE-constrained optimization problems. The application of the ADMM makes it possible to untie the PDE constraints and the nonsmooth regularization terms for iterations. Accordingly, at each iteration, one of the resulting subproblems is a smooth PDE-constrained optimization which can be efficiently solved by PINNs, and the other is a simple nonsmooth optimization problem which usually has a closed-form solution or can be efficiently solved by various standard optimization algorithms or pre-trained neural networks. The ADMM-PINNs algorithmic framework does not require to solve PDEs repeatedly, and it is mesh-free, easy to implement, and scalable to different PDE settings. We validate the efficiency of the ADMM-PINNs algorithmic framework by different prototype applications, including inverse potential problems, source identification in elliptic equations, control constrained optimal control of the Burgers equation, and sparse optimal control of parabolic equations.

[1]  S. Mowlavi,et al.  Optimal control of PDEs using physics-informed neural networks , 2021, J. Comput. Phys..

[2]  Yongcun Song,et al.  A numerical approach to the optimal control of thermally convective flows , 2022, 2211.15302.

[3]  Hang Su,et al.  Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications , 2022, ArXiv.

[4]  S. K. Mahjour,et al.  Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks in Scientific Computing , 2022, ArXiv.

[5]  Hang Su,et al.  Bi-level Physics-Informed Neural Networks for PDE Constrained Optimization using Broyden's Hypergradients , 2022, ArXiv.

[6]  R. Glowinski,et al.  Application of the Alternating Direction Method of Multipliers to Control Constrained Parabolic Optimal Control Problems and Beyond , 2022, Annals of Applied Mathematics.

[7]  Adrian Sandu,et al.  Physics-informed neural networks for PDE-constrained optimization and control , 2022, ArXiv.

[8]  E. Zuazua,et al.  A two-stage numerical approach for the sparse initial source identification of a diffusion-advection equation , 2022, 2202.01589.

[9]  Vincenzo Schiano Di Cola,et al.  Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next , 2022, Journal of Scientific Computing.

[10]  Hyung Ju Hwang,et al.  Solving PDE-constrained Control Problems using Operator Learning , 2021, AAAI.

[11]  George Em Karniadakis,et al.  Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems , 2021, Computer Methods in Applied Mechanics and Engineering.

[12]  Jian Cheng Wong,et al.  CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method , 2021, Computer Methods in Applied Mechanics and Engineering.

[13]  Inanc Senocak,et al.  Physics and equality constrained artificial neural networks: Application to forward and inverse problems with multi-fidelity data fusion , 2021, Journal of Computational Physics.

[14]  Enrique Zuazua,et al.  Control and numerical approximation of fractional diffusion equations , 2021, Numerical Control: Part A.

[15]  Eric Darve,et al.  Physics Constrained Learning for Data-driven Inverse Modeling from Sparse Observations , 2020, J. Comput. Phys..

[16]  Paris Perdikaris,et al.  Fast PDE-constrained optimization via self-supervised operator learning , 2021, ArXiv.

[17]  Xiaoming Yuan,et al.  An ADMM-Newton-CNN numerical approach to a TV model for identifying discontinuous diffusion coefficients in elliptic equations: convex case with gradient observations , 2021, Inverse Problems.

[18]  Paris Perdikaris,et al.  Learning the solution operator of parametric partial differential equations with physics-informed DeepONets , 2021, Science advances.

[19]  Simon W. Funke,et al.  Hybrid FEM-NN models: Combining artificial neural networks with the finite element method , 2021, J. Comput. Phys..

[20]  Steven G. Johnson,et al.  Physics-informed neural networks with hard constraints for inverse design , 2021, SIAM J. Sci. Comput..

[21]  Pouria A. Mistani,et al.  Solving inverse-PDE problems with physics-aware neural networks , 2020, J. Comput. Phys..

[22]  A. Kirsch An Introduction to the Mathematical Theory of Inverse Problems , 1996, Applied Mathematical Sciences.

[23]  A. Wills,et al.  Physics-informed machine learning , 2021, Nature Reviews Physics.

[24]  Siddhartha Mishra,et al.  Iterative Surrogate Model Optimization (ISMO): An active learning algorithm for PDE constrained optimization with deep neural networks , 2020, Computer Methods in Applied Mechanics and Engineering.

[25]  Ryan P. Adams,et al.  Amortized Finite Element Analysis for Fast PDE-Constrained Optimization , 2020, ICML.

[26]  Roland Glowinski,et al.  An ADMM numerical approach to linear parabolic state constrained optimal control problems , 2020, Numerische Mathematik.

[27]  Zhiping Mao,et al.  DeepXDE: A Deep Learning Library for Solving Differential Equations , 2019, AAAI Spring Symposium: MLPS.

[28]  Lexing Ying,et al.  Solving parametric PDE problems with artificial neural networks , 2017, European Journal of Applied Mathematics.

[29]  G. E. Karniadakis,et al.  Variational Physics-Informed Neural Networks For Solving Partial Differential Equations , 2019, ArXiv.

[30]  Xiaoming Yuan,et al.  An Inexact Uzawa Algorithmic Framework for Nonlinear Saddle Point Problems with Applications to Elliptic Optimal Control Problem , 2019, SIAM J. Numer. Anal..

[31]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[32]  Paris Perdikaris,et al.  Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data , 2019, J. Comput. Phys..

[33]  E Weinan,et al.  Machine Learning Approximation Algorithms for High-Dimensional Fully Nonlinear Partial Differential Equations and Second-order Backward Stochastic Differential Equations , 2017, J. Nonlinear Sci..

[34]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[35]  Arnulf Jentzen,et al.  Solving high-dimensional partial differential equations using deep learning , 2017, Proceedings of the National Academy of Sciences.

[36]  Alfio Borzì,et al.  Proximal schemes for parabolic optimal control problems with sparsity promoting cost functionals , 2017, Int. J. Control.

[37]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[38]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[39]  Surya Ganguli,et al.  On the Expressive Power of Deep Neural Networks , 2016, ICML.

[40]  Alfio Borzì,et al.  A LONE code for the sparse control of quantum systems , 2016, Comput. Phys. Commun..

[41]  Weijie Wang,et al.  Alternating Direction Method of Multipliers for Linear Inverse Problems , 2016, SIAM J. Numer. Anal..

[42]  Sabine Fenstermacher,et al.  Estimation Techniques For Distributed Parameter Systems , 2016 .

[43]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[44]  Robert Michael Kirby,et al.  Inverse electrocardiographic source localization of ischemia: An optimization framework and finite element solution , 2013, J. Comput. Phys..

[45]  Michael Hintermüller,et al.  Distributed Optimal Control of the Cahn-Hilliard System Including the Case of a Double-Obstacle Homogeneous Free Energy Density , 2012, SIAM J. Control. Optim..

[46]  Gerd Wachsmuth,et al.  Convergence and regularization results for optimal control problems with sparsity functional , 2011 .

[47]  F. Tröltzsch Optimal Control of Partial Differential Equations: Theory, Methods and Applications , 2010 .

[48]  Georg Stadler,et al.  Elliptic optimal control problems with L1-control cost and applications for the placement of control devices , 2009, Comput. Optim. Appl..

[49]  Stefan Ulbrich,et al.  Optimization with PDE Constraints , 2008, Mathematical modelling.

[50]  Jian-Feng Cai,et al.  Split Bregman Methods and Frame Based Image Restoration , 2009, Multiscale Model. Simul..

[51]  Roland Glowinski,et al.  Exact and Approximate Controllability for Distributed Parameter Systems: A Numerical Approach (Encyclopedia of Mathematics and its Applications) , 2008 .

[52]  Michael Hintermüller,et al.  A SQP-Semismooth Newton-type Algorithm applied to Control of the instationary Navier--Stokes System Subject to Control Constraints , 2006, SIAM J. Optim..

[53]  Albert Tarantola,et al.  Inverse problem theory - and methods for model parameter estimation , 2004 .

[54]  Karl Kunisch,et al.  A comparison of algorithms for control constrained optimal control of the Burgers equation , 2004 .

[55]  Carlos J.S. Alves,et al.  Recovery of cracks using a point-source reciprocity gap function , 2004 .

[56]  M. Nikolova An Algorithm for Total Variation Minimization and Applications , 2004 .

[57]  Xue-Cheng Tai,et al.  Identification of Discontinuous Coefficients in Elliptic Problems Using Total Variation Regularization , 2003, SIAM J. Sci. Comput..

[58]  Richard M. Leahy,et al.  Electromagnetic brain mapping , 2001, IEEE Signal Process. Mag..

[59]  Stefan Volkwein,et al.  Distributed Control Problems for the Burgers Equation , 2001, Comput. Optim. Appl..

[60]  Gene H. Golub,et al.  A Nonlinear Primal-Dual Method for Total Variation-Based Image Restoration , 1999, SIAM J. Sci. Comput..

[61]  Zhiming Chen,et al.  An Augmented Lagrangian Method for Identifying Discontinuous Parameters in Elliptic Systems , 1999 .

[62]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[63]  R. Glowinski,et al.  Exact and approximate controllability for distributed parameter systems , 1995, Acta Numerica.

[64]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[65]  Lyle H. Ungar,et al.  A hybrid neural network‐first principles approach to process modeling , 1992 .

[66]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[67]  W. Ziemer Weakly Differentiable Functions: Sobolev Spaces and Functions of Bounded Variation , 1989 .

[68]  J. Nocedal Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[69]  J. Lions Optimal Control of Systems Governed by Partial Differential Equations , 1971 .

[70]  M. Hestenes Multiplier and gradient methods , 1969 .

[71]  M. Powell A method for nonlinear constraints in minimization problems , 1969 .