A note on solving nonlinear optimization problems in variable precision

This short note considers an efficient variant of the trust-region algorithm with dynamic accuracy proposed by Carter (SIAM J Sci Stat Comput 14(2):368–388, 1993) and by Conn et al. (Trust-region methods. MPS-SIAM series on optimization, SIAM, Philadelphia, 2000) as a tool for very high-performance computing, an area where it is critical to allow multi-precision computations for keeping the energy dissipation under control. Numerical experiments are presented indicating that the use of the considered method can bring substantial savings in objective function’s and gradient’s evaluation “energy costs” by efficiently exploiting multi-precision computations.

[1]  Daniel Brand,et al.  Training Deep Neural Networks with 8-bit Floating Point Numbers , 2018, NeurIPS.

[2]  Jochen W. Schmidt,et al.  Ableitungsfreiebleitungsfreie Verfahren für nichtlineare Optimierungsprobleme , 1970 .

[3]  Tianyi Lin,et al.  Adaptively Accelerating Cubic Regularized Newton's Methods for Convex Optimization via Random Sampling , 2018 .

[4]  Hal Finkel,et al.  Doing Moore with Less - Leapfrogging Moore's Law with Inexactness for Supercomputing , 2016, ArXiv.

[5]  Stefania Bellavia,et al.  A Levenberg–Marquardt method for large nonlinear least-squares problems with dynamic accuracy in functions and gradients , 2018, Numerische Mathematik.

[6]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[7]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[8]  P. Toint,et al.  Partitioned variable metric updates for large structured optimization problems , 1982 .

[9]  Richard G. Carter,et al.  Numerical Experience with a Class of Algorithms for Nonlinear Optimization Using Inexact Function and Gradient Information , 1993, SIAM J. Sci. Comput..

[10]  Hubert Schwetlick,et al.  Computing turning points of curves implicitly defined by nonlinear equations depending on a parameter , 1981, Computing.

[11]  Philippe L. Toint,et al.  WORST-CASE EVALUATION COMPLEXITY AND OPTIMALITY OF SECOND-ORDER METHODS FOR NONCONVEX SMOOTH OPTIMIZATION , 2017, Proceedings of the International Congress of Mathematicians (ICM 2018).

[12]  K. Palem,et al.  Inexactness and a future of computing , 2014, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[13]  Carter,et al.  A worst-case example using linesearch methods for numerical optimization with inexact gradient evaluations , 1994 .

[14]  Mark Horowitz,et al.  Energy-Efficient Floating-Point Unit Design , 2011, IEEE Transactions on Computers.

[15]  S. Bellavia,et al.  Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization , 2018, SIAM J. Optim..

[16]  M. Bartholomew-Biggs,et al.  Some effective methods for unconstrained optimization based on the solution of systems of ordinary differential equations , 1989 .

[17]  Julien Langou,et al.  Accelerating scientific computations with mixed precision algorithms , 2008, Comput. Phys. Commun..

[18]  STEFANIA BELLAVIA,et al.  Adaptive cubic regularization methods with dynamic inexact Hessian information and applications to finite-sum minimization , 2018, IMA Journal of Numerical Analysis.

[19]  Nicholas I. M. Gould,et al.  CUTEst: a Constrained and Unconstrained Testing Environment with safe threads for mathematical optimization , 2013, Computational Optimization and Applications.

[20]  Guangye Li,et al.  The secant/finite difference algorithm for solving sparse nonlinear systems of equations , 1986 .

[21]  Serge Gratton,et al.  Recursive Trust-Region Methods for Multiscale Nonlinear Optimization , 2008, SIAM J. Optim..

[22]  Mark Horowitz,et al.  FPMax: a 106GFLOPS/W at 217GFLOPS/mm2 Single-Precision FPU, and a 43.7GFLOPS/W at 74.6GFLOPS/mm2 Double-Precision FPU, in 28nm UTBB FDSOI , 2016, ArXiv.

[23]  Nicholas John Higham.  The Rise of Multiprecision Arithmetic , 2017, 2017 IEEE 24th Symposium on Computer Arithmetic (ARITH).

[24]  Katya Scheinberg,et al.  Global convergence rate analysis of unconstrained optimization methods based on probabilistic models , 2015, Mathematical Programming.

[25]  P. Toint,et al.  Lancelot: A FORTRAN Package for Large-Scale Nonlinear Optimization (Release A) , 1992 .

[26]  Peng Xu,et al.  Newton-type methods for non-convex optimization under inexact Hessian information , 2017, Math. Program..

[27]  Tianyi Lin,et al.  On Adaptive Cubic Regularized Newton's Methods for Convex Optimization via Random Sampling , 2018 .

[28]  P. Toint,et al.  Global convergence of a class of trust region algorithms for optimization with simple bounds , 1988 .

[29]  Vyacheslav Kungurtsev,et al.  A Subsampling Line-Search Method with Second-Order Results , 2018, INFORMS J. Optim..

[30]  Stefania Bellavia,et al.  Deterministic and stochastic inexact regularization algorithms for nonconvex optimization with optimal complexity , 2018, ArXiv.

[31]  P. Toint,et al.  Performance of a multifrontal scheme for partially separable optimization , 1994 .

[32]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[33]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[34]  Katya Scheinberg,et al.  Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales , 2016, INFORMS Journal on Optimization.

[35]  Clemens Elster,et al.  A method of trust region type for minimizing noisy functions , 1997, Computing.

[36]  Logan Kugler.  Is "good enough" computing good enough? , 2015, Commun. ACM.