A Stochastic Trust Region Algorithm Based on Careful Step Normalization

An algorithm is proposed for solving stochastic and finite-sum minimization problems. Based on a trust region methodology, the algorithm employs normalized steps, at least as long as the norms of t...

[1]  R. F.,et al.  Mathematical Statistics , 1944, Nature.

[2]  K. Chung.  On a Stochastic Approximation Method , 1954 .

[3]  S. Friedman.  On Stochastic Approximations , 1963 .

[4]  E. G. Gladyshev.  On Stochastic Approximation , 1965 .

[5]  T. M. Williams,et al.  Optimizing Methods in Statistics , 1981 .

[6]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[7]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[8]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[9]  Alexander Shapiro,et al.  Stochastic Approximation approach to Stochastic Programming , 2013 .

[10]  Alexander Shapiro,et al.  Lectures on Stochastic Programming: Modeling and Theory , 2009 .

[11]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[12]  T. Sargent,et al.  The multivariate normal distribution , 1989 .

[13]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[14]  Mark W. Schmidt,et al.  Hybrid Deterministic-Stochastic Methods for Data Fitting , 2011, SIAM J. Sci. Comput..

[15]  Jorge Nocedal,et al.  Sample size selection in optimization methods for machine learning , 2012, Math. Program..

[16]  John Langford,et al.  Normalized Online Learning , 2013, UAI.

[17]  Tong Zhang,et al.  Accelerating Stochastic Gradient Descent using Predictive Variance Reduction , 2013, NIPS.

[18]  Saeed Ghadimi,et al.  Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming , 2013, SIAM J. Optim..

[19]  Wotao Yin,et al.  Block Stochastic Gradient Iteration for Convex and Nonconvex Optimization , 2014, SIAM J. Optim..

[20]  Léon Bottou,et al.  A Lower Bound for the Optimization of Finite Sums , 2014, ICML.

[21]  Shai Shalev-Shwartz,et al.  Beyond Convexity: Stochastic Quasi-Convex Optimization , 2015, NIPS.

[22]  Guanghui Lan,et al.  Stochastic Block Mirror Descent Methods for Nonsmooth and Stochastic Optimization , 2013, SIAM J. Optim..

[23]  Mark W. Schmidt,et al.  Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition , 2016, ECML/PKDD.

[24]  Jeffrey Larson,et al.  Stochastic derivative-free optimization using a trust region framework , 2016, Comput. Optim. Appl..

[25]  Yue Xie,et al.  SI-ADMM: A stochastic inexact ADMM framework for resolving structured stochastic convex programs , 2016, 2016 Winter Simulation Conference (WSC).

[26]  Saeed Ghadimi,et al.  Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization , 2013, Mathematical Programming.

[27]  Rui Shi,et al.  A Stochastic Trust Region Algorithm , 2017 .

[28]  Katya Scheinberg,et al.  Stochastic optimization using a trust-region method and random models , 2015, Mathematical Programming.

[29]  Optimal Smoothed Variable Sample-size Accelerated Proximal Methods for Structured Nonsmooth Stochastic Convex Programs , 2018 .

[30]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[31]  Jinlong Lei,et al.  A Randomized Block Proximal Variable Sample-size Stochastic Gradient Method for Composite Nonconvex Stochastic Optimization , 2018 .

[32]  Alejandro Jofré,et al.  On variance reduction for stochastic smooth convex optimization with multiplicative noise , 2017, Math. Program..