A gradient descent akin method for constrained optimization: algorithms and applications

We present a first-order method for solving constrained optimization problems. The method is derived from our previous work, a modified search direction method inspired by singular value decomposition. In this work, we simplify its computational framework to a ``gradient descent akin'' method (GDAM), i.e., the search direction is computed using a linear combination of the negative and normalized objective and constraint gradient. We give fundamental theoretical guarantees on the global convergence of the method. This work focuses on the algorithms and applications of GDAM. We present computational algorithms that adapt common strategies for the gradient descent method. We demonstrate the potential of the method using two engineering applications, shape optimization and sensor network localization. When practically implemented, GDAM is robust and very competitive in solving the considered large and challenging optimization problems.

[1]  J. Martins,et al.  Engineering Design Optimization , 2021 .

[2]  Antonis Papachristodoulou,et al.  Chordal and factor-width decompositions for scalable semidefinite and polynomial optimization , 2021, Annu. Rev. Control..

[3]  Michael A. Saunders,et al.  Large-scale Optimization with Linear Equality Constraints using Reduced Compact Representation , 2021, SIAM J. Sci. Comput..

[4]  F. Barthold,et al.  Shape optimization of the X0-specimen: theory, numerical simulation and experimental verification , 2020, Computational Mechanics.

[5]  Brendan O'Donoghue Operator Splitting for a Homogeneous Embedding of the Linear Complementarity Problem , 2020, SIAM J. Optim..

[6]  G. Allaire,et al.  Null space gradient flows for constrained optimization with applications to shape optimization , 2020, ESAIM: Control, Optimisation and Calculus of Variations.

[7]  Grigorios Koulouras,et al.  Applications of Wireless Sensor Networks: An Up-to-Date Survey , 2020, Applied System Innovation.

[8]  Simon W. Funke,et al.  Automatic shape derivatives for transient PDEs in FEniCS and Firedrake , 2020, 2001.10058.

[9]  Roland Wüchner,et al.  A partitioned scheme for adjoint shape sensitivity analysis of fluid–structure interactions involving non-matching meshes , 2019, Optim. Methods Softw..

[10]  Amir Ali Ahmadi,et al.  A Survey of Recent Scalability Improvements for Semidefinite Programming with Applications in Machine Learning, Control, and Robotics , 2019, Annu. Rev. Control. Robotics Auton. Syst..

[11]  Long Chen,et al.  A modified search direction method for inequality constrained optimization problems using the singular-value decomposition of normalized response gradients , 2019, Structural and Multidisciplinary Optimization.

[12]  Radu Ioan Bot,et al.  A primal-dual dynamical approach to structured convex minimization problems , 2019, 1905.08290.

[13]  Michael I. Jordan DYNAMICAL, SYMPLECTIC AND STOCHASTIC PERSPECTIVES ON GRADIENT-BASED OPTIMIZATION , 2019, Proceedings of the International Congress of Mathematicians (ICM 2018).

[14]  Daniel Luft,et al.  Efficient Techniques for Shape Optimization with Variational Inequalities Using Adjoints , 2019, SIAM J. Optim..

[15]  Kresimir Mihic,et al.  Managing randomization in the multi-block alternating direction method of multipliers for quadratic optimization , 2019, Math. Program. Comput..

[16]  Peter Eberhard,et al.  Geometric element parameterization and parametric model order reduction in finite element based shape optimization , 2018, Computational Mechanics.

[17]  R. Michael Buehrer,et al.  Collaborative Sensor Network Localization: Algorithms and Practical Issues , 2018, Proceedings of the IEEE.

[18]  M. Montaz Ali,et al.  A Trajectory-Based Method for Constrained Nonlinear Optimization Problems , 2018, J. Optim. Theory Appl..

[19]  Sanjeev Arora,et al.  On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization , 2018, ICML.

[20]  Stephen P. Boyd,et al.  OSQP: an operator splitting solver for quadratic programs , 2017, Mathematical Programming Computation.

[21]  Kim-Chuan Toh,et al.  SDPNAL+: A Matlab software for semidefinite programming with bound constraints (version 1.0) , 2017, Optim. Methods Softw..

[22]  S. Gaubert,et al.  Log-Barrier Interior Point Methods Are Not Strongly Polynomial , 2017, SIAM Journal on applied algebra and geometry.

[23]  Danda B. Rawat,et al.  Advances on localization techniques for wireless sensor networks: A survey , 2016, Comput. Networks.

[24]  Michal Kocvara,et al.  Primal-Dual Interior Point Multigrid Method for Topology Optimization , 2016, SIAM J. Sci. Comput..

[25]  Tim A. Albring,et al.  Efficient Aerodynamic Design using the Discrete Adjoint Method in SU2 , 2016 .

[26]  Dimitri P. Bertsekas,et al.  Convex Optimization Algorithms , 2015 .

[27]  Graeme J. Kennedy Large-Scale Multimaterial Topology Optimization for Additive Manufacturing , 2015 .

[28]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[29]  V. Schulz,et al.  Towards a Lagrange–Newton Approach for PDE Constrained Shape Optimization , 2014, 1405.3266.

[30]  Graeme J. Kennedy,et al.  Scalable Parallel Approach for High-Fidelity Steady-State Aeroelastic Analysis and Adjoint Derivative Computations , 2014 .

[31]  V. Schulz,et al.  Three-Dimensional Large-Scale Aerodynamic Shape Optimization Based on Shape Calculus , 2013 .

[32]  Joaquim R. R. A. Martins,et al.  Multidisciplinary design optimization: A survey of architectures , 2013 .

[33]  Jacek Gondzio,et al.  Interior point methods 25 years later , 2012, Eur. J. Oper. Res..

[34]  Emmanuel J. Candès,et al.  Adaptive Restart for Accelerated Gradient Schemes , 2012, Foundations of Computational Mathematics.

[35]  Claudia Schillings,et al.  Efficient shape optimization for certain and uncertain aerodynamic design , 2011 .

[36]  Joaquim R. R. A. Martins,et al.  pyOpt: a Python-based object-oriented framework for nonlinear constrained optimization , 2011, Structural and Multidisciplinary Optimization.

[37]  Tomi Räty,et al.  Survey on Contemporary Remote Surveillance Systems for Public Safety , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[38]  Zhi-Quan Luo,et al.  Semidefinite Relaxation of Quadratic Optimization Problems , 2010, IEEE Signal Processing Magazine.

[39]  Jason E. Hicken,et al.  Aerodynamic Optimization Algorithm with Integrated Geometry Parameterization and Mesh Movement , 2010 .

[40]  Stephen P. Boyd,et al.  Convex Optimization , 2010, IEEE Transactions on Automatic Control.

[41]  Jože Korelc,et al.  Automation of primal and sensitivity analysis of transient coupled problems , 2009 .

[42]  Harbir Antil,et al.  Adaptive Path Following Primal Dual Interior Point Methods for Shape Optimization of Linear and Nonlinear Stokes Flow Problems , 2009, LSSC.

[43]  Masakazu Kojima,et al.  Exploiting Sparsity in SDP Relaxation for Sensor Network Localization , 2009, SIAM J. Optim..

[44]  Stephen P. Boyd,et al.  Further Relaxations of the Semidefinite Programming Approach to Sensor Network Localization , 2008, SIAM J. Optim..

[45]  Yinyu Ye,et al.  Algorithm 875: DSDP5—software for semidefinite programming , 2008, TOMS.

[46]  Naomi Ehrich Leonard,et al.  Collective Motion, Sensor Networks, and Ocean Sampling , 2007, Proceedings of the IEEE.

[47]  Harbir Antil,et al.  Path-following primal-dual interior-point methods for shape optimization of stationary flow problems , 2007, J. Num. Math..

[48]  Jorge Cortés,et al.  Finite-time convergent gradient flows with applications to network consensus , 2006, Autom..

[49]  Yinyu Ye,et al.  Semidefinite programming based algorithms for sensor network localization , 2006, TOSN.

[50]  D. Puccinelli,et al.  Wireless sensor networks: applications and challenges of ubiquitous sensing , 2005, IEEE Circuits and Systems Magazine.

[51]  Ling-Jyh Chen,et al.  Reliable sensor networks for planet exploration , 2005, Proceedings. 2005 IEEE Networking, Sensing and Control, 2005..

[52]  Anthony Man-Cho So,et al.  Theory of semidefinite programming for Sensor Network Localization , 2005, SODA '05.

[53]  Yinyu Ye,et al.  Semidefinite programming for ad hoc wireless sensor network localization , 2004, Third International Symposium on Information Processing in Sensor Networks, 2004. IPSN 2004.

[54]  G. Allaire,et al.  Structural optimization using sensitivity analysis and a level-set method , 2004 .

[55]  Michael Stingl,et al.  PENNON: A code for convex nonlinear and semidefinite programming , 2003, Optim. Methods Softw..

[56]  X. Q. Yang,et al.  A Unified Gradient Flow Approach to Constrained Nonlinear Optimization Problems , 2003, Comput. Optim. Appl..

[57]  Anders Forsgren,et al.  Interior Methods for Nonlinear Optimization , 2002, SIAM Rev..

[58]  Ian F. Akyildiz,et al.  Wireless sensor networks: a survey , 2002, Comput. Networks.

[59]  G. Dias,et al.  Shape structural optimization with an interior point nonlinear programming algorithm , 2000 .

[60]  Volker Schulz,et al.  Interior point multigrid methods for topology optimization , 2000 .

[61]  Alex M. Andrew Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science (2nd edition) , 2000 .

[62]  H. Attouch,et al.  THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEM , 2000 .

[63]  T. Vicsek,et al.  Collective Motion , 1999, physics/9902023.

[64]  Florian Jarre,et al.  Optimal Truss Design by Interior-Point Methods , 1998, SIAM J. Optim..

[65]  David P. Williamson,et al.  Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming , 1995, JACM.

[66]  Farid Alizadeh,et al.  Interior Point Methods in Semidefinite Programming with Applications to Combinatorial Optimization , 1995, SIAM J. Optim..

[67]  N. Stander,et al.  On the robustness and efficiency of the SAM algorithm for structural optimization , 1995 .

[68]  E. D. Klerk,et al.  A feasible descent cone method for linearly constrained minimization problems , 1994 .

[69]  U. Helmke,et al.  Optimization and Dynamical Systems , 1994, Proceedings of the IEEE.

[70]  Jan A. Snyman,et al.  A new first‐order interior feasible direction method for structural optimization , 1993 .

[71]  R. Haftka,et al.  On options for interdisciplinary analysis and design optimization , 1992 .

[72]  D. Bayer,et al.  The nonlinear geometry of linear programming. I. Affine and projective scaling trajectories , 1989 .

[73]  Jasbir S. Arora,et al.  Introduction to Optimum Design , 1988 .

[74]  J. Snyman,et al.  A multi-start global minimization algorithm with dynamic search trajectories , 1987 .

[75]  K. Svanberg The method of moving asymptotes—a new method for structural optimization , 1987 .

[76]  A. Griewank Generalized descent for global optimization , 1981 .

[77]  C. Botsaris Differential gradient methods , 1978 .

[78]  K. Linkwitz,et al.  Einige Bemerkungen zur Berechnung von vorgespannten Seilnetzkonstruktionen , 1971 .

[79]  M. J. D. Powell,et al.  Nonlinear Programming—Sequential Unconstrained Minimization Techniques , 1969 .

[80]  J. B. Rosen The Gradient Projection Method for Nonlinear Programming. Part I. Linear Constraints , 1960 .

[81]  Michael I. Jordan,et al.  A Lyapunov Analysis of Accelerated Methods in Optimization , 2021, J. Mach. Learn. Res..

[82]  Nikolaos Ploskas,et al.  Interior Point Methods , 2017 .

[83]  Phillipp Kaestner,et al.  Linear And Nonlinear Programming , 2016 .

[84]  R. Rao,et al.  Jaya: A simple and new optimization algorithm for solving constrained and unconstrained optimization problems , 2016 .

[85]  Michael R Wisnom,et al.  56th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference , 2015 .

[86]  John T. Hwang A Modular Approach to Large-scale Design Optimization of Aerospace Systems. , 2015 .

[87]  Joaquim R. R. A. Martins,et al.  Comparison of inexact- and quasi-Newton algorithms for aerodynamic shape optimization , 2015 .

[88]  Kai-Uwe Bletzinger,et al.  The Vertex Morphing method for node-based shape optimization , 2014 .

[89]  Uwe Naumann,et al.  A Discrete Adjoint Model for OpenFOAM , 2013, ICCS.

[90]  Makoto Yamashita,et al.  Latest Developments in the SDPA Family for Solving Large-Scale SDPs , 2012 .

[91]  F. Verhulst Differential Equations and Dynamical Systems , 2012 .

[92]  Kai-Uwe Bletzinger,et al.  Free Shape Optimal Design of Structures , 2011 .

[93]  Y. Ye,et al.  A Distributed Method for Solving Semidefinite Programs Arising from Ad Hoc Wireless Sensor Network Localization , 2006 .

[94]  Jing J. Liang,et al.  Problem Deflnitions and Evaluation Criteria for the CEC 2006 Special Session on Constrained Real-Parameter Optimization , 2006 .

[95]  Yinyu Ye,et al.  DSDP5: Software for Semidefinite Programming , 2005 .

[96]  K. N. Seetharamu,et al.  The Finite Element Method , 2005 .

[97]  J. Lofberg,et al.  YALMIP : a toolbox for modeling and optimization in MATLAB , 2004, 2004 IEEE International Conference on Robotics and Automation (IEEE Cat. No.04CH37508).

[98]  Raino A. E. Mäkinen,et al.  Introduction to shape optimization - theory, approximation, and computation , 2003, Advances in design and control.

[99]  Sven Leyffer,et al.  Nonlinear programming without a penalty function , 2002, Math. Program..

[100]  Knud D. Andersen,et al.  The Mosek Interior Point Optimizer for Linear Programming: An Implementation of the Homogeneous Algorithm , 2000 .

[101]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[102]  Y. Ye,et al.  Semidefinite programming relaxations of nonconvex quadratic optimization , 2000 .

[103]  Michael M. Kostreva,et al.  Methods of Feasible Directions: A Review , 2000 .

[104]  Jorge Nocedal,et al.  A trust region method based on interior point techniques for nonlinear programming , 2000, Math. Program..

[105]  Walter Murray,et al.  An efficient gradient flow method for unconstrained optimization , 1998 .

[106]  Y. Ye Interior point algorithms - theory and analysis , 1998, Wiley-Interscience series in discrete mathematics and optimization.

[107]  G. Allaire,et al.  Shape optimization by the homogenization method , 1997 .

[108]  Immo Diener,et al.  Trajectory Methods in Global Optimization , 1995 .

[109]  Jan Sokolowski,et al.  Introduction to shape optimization , 1992 .

[110]  J. Lagarias The nonlinear geometry of linear programming. III. Projective Legendre transform coordinates and Hilbert geometry , 1990 .

[111]  Y. Nesterov A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[112]  G. Zoutendijk,et al.  Methods of feasible directions : a study in linear and non-linear programming , 1960 .