Optimization with learning-informed differential equation constraints and its applications

Inspired by applications in optimal control of semilinear elliptic partial differential equations and physics-integrated imaging, differential equation constrained optimization problems with constituents that are only accessible through data-driven techniques are studied. A particular focus is on the analysis and on numerical methods for problems with machine-learned components. For a rather general context, an error analysis is provided, and particular properties resulting from artificial neural network based approximations are addressed. Moreover, for each of the two inspiring applications analytical details are presented and numerical results are provided.

[1]  Hector O. Fattorini,et al.  Infinite Dimensional Optimization and Control Theory: References , 1999 .

[2]  Sairam Geethanath,et al.  Magnetic Resonance Fingerprinting Reconstruction via Spatiotemporal Convolutional Neural Networks , 2018, MLMIR@MICCAI.

[3]  Martin Hanke,et al.  The regularizing Levenberg-Marquardt scheme is of optimal order , 2010 .

[4]  Guy Van den Broeck,et al.  Counterexample-Guided Learning of Monotonic Neural Networks , 2020, NeurIPS.

[5]  J. Duerk,et al.  Magnetic Resonance Fingerprinting , 2013, Nature.

[6]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[7]  Jonas Adler,et al.  Solving ill-posed inverse problems using iterative deep neural networks , 2017, ArXiv.

[8]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[9]  Michael Hintermüller,et al.  Quantitative Magnetic Resonance Imaging: From Fingerprinting to Integrated Physics-Based Models , 2019, SIAM J. Imaging Sci..

[10]  Dan Tiba,et al.  Optimization of Elliptic Systems , 2006 .

[11]  Kim C. Border,et al.  Infinite Dimensional Analysis: A Hitchhiker’s Guide , 1994 .

[12]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[13]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[14]  Gitta Kutyniok,et al.  Error bounds for approximations with deep ReLU neural networks in $W^{s, p}$ norms , 2019, Analysis and Applications.

[15]  Eldad Haber,et al.  Stable architectures for deep neural networks , 2017, ArXiv.

[16]  J. Lions.  Optimal Control of Systems Governed by Partial Differential Equations , 1971 .

[17]  Ramesh Raskar,et al.  Designing Neural Network Architectures using Reinforcement Learning , 2016, ICLR.

[18]  Yonina C. Eldar,et al.  Low rank magnetic resonance fingerprinting , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[19]  G. D. Maso,et al.  An Introduction to-convergence , 1993 .

[20]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[21]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[22]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  G. Burton.  Sobolev Spaces , 2013 .

[24]  Qiang Liu,et al.  Certified Monotonic Neural Networks , 2020, NeurIPS.

[25]  Andrea Braides.  Local Minimization, Variational Evolution and Γ-Convergence , 2013 .

[26]  Razvan Pascanu,et al.  Sobolev Training for Neural Networks , 2017, NIPS.

[27]  J. Zowe,et al.  Regularity and stability for the mathematical programming problem in Banach spaces , 1979 .

[28]  P. Malliavin.  Infinite dimensional analysis , 1993 .

[29]  Michael Hintermüller,et al.  Mesh independence and fast local convergence of a primal-dual active-set method for mixed control-state constrained elliptic control problems , 2007 .

[30]  Pierre Vandergheynst,et al.  A Compressed Sensing Framework for Magnetic Resonance Fingerprinting , 2013, SIAM J. Imaging Sci..

[31]  Leon Bungert,et al.  CLIP: Cheap Lipschitz Training of Neural Networks , 2021, SSVM.

[32]  I. Gavrilyuk.  Book Review: Variational analysis in Sobolev and BV spaces , 2007 .

[33]  Arnulf Jentzen,et al.  Solving high-dimensional partial differential equations using deep learning , 2017, Proceedings of the National Academy of Sciences.

[34]  Michael Ulbrich,et al.  A mesh-independence result for semismooth Newton methods , 2004, Math. Program..

[35]  Kailiang Wu,et al.  Data Driven Governing Equations Approximation Using Deep Neural Networks , 2018, J. Comput. Phys..

[36]  Garrison W. Cottrell,et al.  ReZero is All You Need: Fast Convergence at Large Depth , 2020, UAI.

[37]  Simon R. Arridge,et al.  Solving inverse problems using data-driven models , 2019, Acta Numerica.

[38]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[39]  Jens Flemming,et al.  Convergence rate analysis of Tikhonov regularization for nonlinear ill-posed problems with noisy operators , 2012 .

[40]  R. Schnabel,et al.  A view of unconstrained optimization , 1989 .

[41]  D. Louis Collins,et al.  Design and construction of a realistic digital brain phantom , 1998, IEEE Transactions on Medical Imaging.

[42]  S. Dirkse,et al.  The path solver: a nommonotone stabilization scheme for mixed complementarity problems , 1995 .

[43]  E Weinan,et al.  A Proposal on Machine Learning via Dynamical Systems , 2017, Communications in Mathematics and Statistics.

[44]  K. Scheffler.  A pictorial description of steady-states in rapid magnetic resonance imaging , 1999 .

[45]  Karl Kunisch,et al.  Feasible and Noninterior Path-Following in Constrained Minimization with Low Multiplier Regularity , 2006, SIAM J. Control. Optim..

[46]  G. M.,et al.  Partial Differential Equations I , 2023, Applied Mathematical Sciences.

[47]  T. Sideris.  Ordinary Differential Equations and Dynamical Systems , 2013 .

[48]  Ivan P. Gavrilyuk,et al.  Variational analysis in Sobolev and BV spaces , 2007, Math. Comput..

[49]  F. Tröltzsch.  Optimal Control of Partial Differential Equations: Theory, Methods and Applications , 2010 .

[50]  Kazufumi Ito,et al.  The Primal-Dual Active Set Strategy as a Semismooth Newton Method , 2002, SIAM J. Optim..

[51]  W. W. Hansen,et al.  Nuclear Induction , 2011 .

[52]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[53]  Marina Velikova,et al.  Monotone and Partially Monotone Neural Networks , 2010, IEEE Transactions on Neural Networks.

[54]  Daniel Ralph,et al.  Global Convergence of Damped Newton's Method for Nonsmooth Equations via the Path Search , 1994, Math. Oper. Res..