A Derivative-Free Trust-Region Algorithm for the Optimization of Functions Smoothed via Gaussian Convolution Using Adaptive Multiple Importance Sampling

In this paper we consider the optimization of a functional $F$ defined as the convolution of a function $f$ with a Gaussian kernel. We propose this type of objective function for the optimization of the output of complex computational simulations, which often present some form of deterministic noise and need to be smoothed for the results to be meaningful. We introduce a derivative-free algorithm that computes trial points from the minimization of a regression model of the noisy function $f$ over a trust region. The regression model is constructed from function values at sample points that are chosen randomly around iterates and trial points of the algorithm. The weights given to the individual sample points in the regression problem are obtained according to an adaptive multiple importance sampling strategy. This has two advantages. First, it makes it possible to reuse all noisy function values collected over the course of the optimization. Second, the resulting regression model converges to the second-o...

[1]  Raghu Pasupathy,et al.  ASTRO-DF: A Class of Adaptive Sampling Trust-Region Algorithms for Derivative-Free Stochastic Optimization , 2016, SIAM J. Optim..

[2]  Stefan M. Wild,et al.  Estimating Derivatives of Noisy Simulations , 2012, TOMS.

[3]  Stefan M. Wild,et al.  Estimating Computational Noise , 2011, SIAM J. Sci. Comput..

[4]  Katya Scheinberg,et al.  Geometry of interpolation sets in derivative free optimization , 2007, Math. Program..

[5]  J. Kiefer,et al.  Stochastic Estimation of the Maximum of a Regression Function , 1952 .

[6]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[7]  D. Mayne,et al.  Nondifferential optimization via adaptive smoothing , 1984 .

[8]  Carl Tim Kelley,et al.  Iterative methods for optimization , 1999, Frontiers in applied mathematics.

[9]  W. Cleveland,et al.  Smoothing by Local Regression: Principles and Methods , 1996 .

[10]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[11]  Dimitri P. Bertsekas,et al.  The effect of deterministic noise in subgradient methods , 2010, Math. Program..

[12]  Katya Scheinberg,et al.  A derivative free optimization algorithm in practice , 1998 .

[13]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[14]  Katya Scheinberg,et al.  Stochastic optimization using a trust-region method and random models , 2015, Mathematical Programming.

[15]  Katya Scheinberg,et al.  Self-Correcting Geometry in Model-Based Algorithms for Derivative-Free Unconstrained Optimization , 2010, SIAM J. Optim..

[16]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[17]  Jean-Marie Cornuet,et al.  Adaptive Multiple Importance Sampling , 2009, 0907.1254.

[18]  J. J. More,et al.  Global smoothing and continuation for large-scale molecular optimization , 1995 .

[19]  Katya Scheinberg,et al.  Methodologies and software for derivative-free optimization , 2017 .

[20]  Dominique Pelletier,et al.  On Sensitivity Analysis for Problems with Numerical Noise , 2002 .

[21]  J. Spall.  Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , 1992 .

[22]  David Lyzenga,et al.  Surface-Wavefield Estimation From Coherent Marine Radars , 2010, IEEE Geoscience and Remote Sensing Letters.

[23]  Michael C. Ferris,et al.  Adaptation of the Uobyqa Algorithm for Noisy Functions , 2006, Proceedings of the 2006 Winter Simulation Conference.

[24]  Andreas Wächter,et al.  Uniform Convergence of Sample Average Approximation with Adaptive Importance Sampling , 2015 .

[25]  Jeremy Staum,et al.  Sample Average Approximation with Adaptive Importance Sampling , 2017 .

[26]  A. Owen,et al.  Safe and Effective Importance Sampling , 2000 .

[27]  Katya Scheinberg,et al.  Convergence of Trust-Region Methods Based on Probabilistic Models , 2013, SIAM J. Optim..

[28]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[29]  Bouhari Arouna,et al.  Adaptative Monte Carlo Method, A Variance Reduction Technique , 2004, Monte Carlo Methods Appl..

[30]  Petar M. Djuric,et al.  Adaptive Importance Sampling: The past, the present, and the future , 2017, IEEE Signal Processing Magazine.

[31]  Robert Hooke,et al.  `` Direct Search'' Solution of Numerical and Statistical Problems , 1961, JACM.

[32]  J. Blanchet,et al.  Convergence Rate Analysis of a Stochastic Trust Region Method for Nonconvex Optimization , 2016 .

[33]  K. E. Vugrin.  On the Effects of Noise on Parameter Identification Optimization Problems , 2005 .

[34]  Jeffrey Larson,et al.  Stochastic derivative-free optimization using a trust region framework , 2016, Comput. Optim. Appl..

[35]  Reuven Y. Rubinstein,et al.  Nondifferentiable optimization via smooth approximation: General analytical approach , 1992, Ann. Oper. Res..

[36]  Hong Wan,et al.  Stochastic Trust-Region Response-Surface Method (STRONG) - A New Response-Surface Framework for Simulation Optimization , 2013, INFORMS J. Comput..

[37]  Stefan M. Wild,et al.  Benchmarking Derivative-Free Optimization Algorithms , 2009, SIAM J. Optim..

[38]  G. Box,et al.  On the Experimental Attainment of Optimum Conditions , 1951 .

[39]  Katya Scheinberg,et al.  Global Convergence of General Derivative-Free Trust-Region Algorithms to First- and Second-Order Critical Points , 2009, SIAM J. Optim..

[40]  Katya Scheinberg,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[41]  Bernardetta Addis,et al.  Local optima smoothing for global optimization , 2005, Optim. Methods Softw..

[42]  Hossein Mobahi,et al.  Seeing through the blur , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  G. Terrell.  Statistical theory and computational aspects of smoothing , 1997 .

[44]  Yi Ma,et al.  Gaussian Smoothing and Asymptotic Convexity , 2012 .

[45]  Jorge Nocedal,et al.  On the geometry phase in model-based algorithms for derivative-free optimization , 2009, Optim. Methods Softw..

[46]  Hans Bruun Nielsen,et al.  UCTP - Test Problems for Unconstrained Optimization , 2000 .

[47]  Andreas Wächter,et al.  UNIFORM CONVERGENCE OF SAMPLE AVERAGE APPROXIMATION WITH ADAPTIVE MULTIPLE IMPORTANCE SAMPLING , 2018, 2018 Winter Simulation Conference (WSC).

[48]  H. Kushner,et al.  Stochastic Approximation and Recursive Algorithms and Applications , 2003 .

[49]  Philip N. Strenski,et al.  Gradient-based optimization of custom circuits using a static-timing formulation , 1999, DAC '99.

[50]  Jeffrey Larson,et al.  Derivative-Free Optimization of Expensive Functions with Computational Error Using Weighted Regression , 2013, SIAM J. Optim..

[51]  Ronald A. Rohrer,et al.  Piecewise approximate circuit simulation , 1991, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[52]  Reuven Y. Rubinstein,et al.  Smoothed Functionals in Stochastic Optimization , 1983, Math. Oper. Res..

[53]  Jorge Nocedal,et al.  Wedge trust region methods for derivative free optimization , 2002, Math. Program..

[54]  D. Gleich.  TRUST REGION METHODS , 2017 .

[55]  Leonidas J. Guibas,et al.  Optimally combining sampling techniques for Monte Carlo rendering , 1995, SIGGRAPH.

[56]  Robert F. Beck,et al.  Seakeeping computations using double-body basis flows , 2010 .

[57]  L. Shepp.  Probability Essentials , 2002 .

[58]  Okey Nwogu,et al.  Interaction of finite-amplitude waves with vertically sheared current fields , 2009, Journal of Fluid Mechanics.

[59]  M. J. D. Powell,et al.  UOBYQA: unconstrained optimization by quadratic approximation , 2002, Math. Program..

[60]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[61]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[62]  Paul Glasserman,et al.  Monte Carlo Methods in Financial Engineering , 2003 .

[63]  Katya Scheinberg,et al.  Computation of sparse low degree interpolating polynomials and their application to derivative-free optimization , 2012, Mathematical Programming.