Noise-Tolerant Optimization Methods for the Solution of a Robust Design Problem

The development of nonlinear optimization algorithms capable of performing reliably in the presence of noise has garnered considerable attention lately. This paper advocates for strategies to create noise-tolerant nonlinear optimization algorithms by adapting classical deterministic methods. These adaptations follow certain design guidelines described here, which make use of estimates of the noise level in the problem. The application of our methodology is illustrated by the development of a line search gradient projection method, which is tested on an engineering design problem. It is shown that a new self-calibrated line search and noise-aware finite-difference techniques are effective even in the high noise regime. Numerical experiments investigate the resiliency of key algorithmic components. A convergence analysis of the line search gradient projection method establishes convergence to a neighborhood of the solution.

[1]  Shigeng Sun,et al.  Stochastic Ratios Tracking Algorithm for Large Scale Machine Learning Problems , 2023, ArXiv.

[2]  K. Scheinberg,et al.  Sample Complexity Analysis for Adaptive Optimization Algorithms with Stochastic Oracles , 2023, 2303.06838.

[3]  Michael W. Mahoney,et al.  Fully Stochastic Trust-Region Sequential Quadratic Programming for Equality-Constrained Optimization Problems , 2022, 2211.15943.

[4]  J. Nocedal,et al.  On the numerical performance of finite-difference-based methods for derivative-free optimization , 2022, Optim. Methods Softw..

[5]  K. Scheinberg,et al.  First- and second-order high probability complexity bounds for trust-region methods with noisy oracles , 2022, Mathematical Programming.

[6]  N. I. M. Gould,et al.  An adaptive regularization algorithm for unconstrained optimization with inexact function and derivatives values , 2021, 2111.14098.

[7]  J. Nocedal,et al.  Adaptive Finite-Difference Interval Estimation for Noisy Derivative-Free Optimization , 2021, SIAM J. Sci. Comput..

[8]  Jorge Nocedal,et al.  Constrained Optimization in the Presence of Noise , 2021, SIAM J. Optim..

[9]  Daniel P. Robinson,et al.  A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear-Equality-Constrained Optimization with Rank-Deficient Jacobians , 2021, Mathematics of Operations Research.

[10]  K. Scheinberg,et al.  High Probability Complexity Bounds for Adaptive Step Search Based on Stochastic Oracles , 2021, 2106.06454.

[11]  Stefania Bellavia,et al.  The Impact of Noise on Evaluation Complexity: The Deterministic Trust-Region Case , 2021, Journal of Optimization Theory and Applications.

[12]  P. Toint,et al.  Strong Evaluation Complexity of An Inexact Trust-Region Algorithm for Arbitrary-Order Unconstrained Nonconvex Optimization. , 2020, 2011.00854.

[13]  Jorge Nocedal,et al.  A Noise-Tolerant Quasi-Newton Algorithm for Unconstrained Optimization , 2020, SIAM J. Optim..

[14]  Daniel P. Robinson,et al.  Sequential Quadratic Optimization for Nonlinear Equality Constrained Stochastic Optimization , 2020, SIAM J. Optim..

[15]  Xingshi He,et al.  Introduction to Optimization , 2015, Optimization for Chemical and Biochemical Engineering.

[16]  Albert S. Berahas,et al.  Global Convergence Rate Analysis of a Generic Line Search Algorithm with Noise , 2019, SIAM J. Optim..

[17]  Jorge Nocedal,et al.  Analysis of the BFGS Method with Errors , 2019, SIAM J. Optim..

[18]  P. Gill,et al.  Practical optimization , 2019 .

[19]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[20]  Jorge Nocedal,et al.  Derivative-Free Optimization of Noisy Functions via Quasi-Newton Methods , 2018, SIAM J. Optim..

[21]  Rui Shi,et al.  A Stochastic Trust Region Algorithm Based on Careful Step Normalization , 2017, INFORMS J. Optim..

[22]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[23]  D. Bertsekas.  Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey , 2015, ArXiv.

[24]  Katya Scheinberg,et al.  Stochastic optimization using a trust-region method and random models , 2015, Mathematical Programming.

[25]  Leo Wai-Tsun Ng,et al.  Multifidelity approaches for optimization under uncertainty , 2014 .

[26]  Alexander Shapiro,et al.  Stochastic Approximation approach to Stochastic Programming , 2013 .

[27]  Stefan M. Wild,et al.  Estimating Derivatives of Noisy Simulations , 2012, TOMS.

[28]  Stefan M. Wild,et al.  Estimating Computational Noise , 2011, SIAM J. Sci. Comput..

[29]  Michael A. Saunders,et al.  SNOPT: An SQP Algorithm for Large-Scale Constrained Optimization , 2002, SIAM J. Optim..

[30]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[31]  S. Granville.  Optimal reactive dispatch through interior point methods , 1994 .

[32]  P. Toint,et al.  Lancelot: A FORTRAN Package for Large-Scale Nonlinear Optimization (Release A) , 1992 .

[33]  J. G. Saw,et al.  Chebyshev Inequality With Estimated Mean and Variance , 1984 .

[34]  P. Gill,et al.  Computing Forward-Difference Intervals for Numerical Optimization , 1983 .

[35]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[36]  Stephen M. Robinson,et al.  Perturbed Kuhn-Tucker points and rates of convergence for a class of nonlinear-programming algorithms , 1974, Math. Program..

[37]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[38]  C. Elster,et al.  A trust region method for the optimization of noisy functions , 2007 .

[39]  Richard W. Hamming,et al.  Introduction to Applied Numerical Analysis. , 1971 .

[40]  P. Pachowicz,et al.  A NOISE-TOLERANT , 2022 .