Stochastic momentum methods for non-convex learning without bounded assumptions.

[1]  Aytaç Altan,et al.  Artificial Intelligence-Based Robust Hybrid Algorithm Design and Implementation for Real-Time Detection of Plant Diseases in Agricultural Environments , 2022, Biology.

[2]  Dongpo Xu,et al.  On hyper-parameter selection for guaranteed convergence of RMSProp , 2022, Cognitive Neurodynamics.

[3]  Minghui Zhu,et al.  An adaptive polyak heavy-ball method , 2022, Machine Learning.

[4]  Dongpo Xu,et al.  Last-iterate convergence analysis of stochastic momentum methods for neural networks , 2022, Neurocomputing.

[5]  Jun Liu,et al.  On Almost Sure Convergence Rates of Stochastic Gradient Methods , 2022, COLT.

[6]  Dongpo Xu,et al.  SGD-rα: A real-time α-suffix averag , 2022, Neurocomputing.

[7]  Tao Sun,et al.  Sign Stochastic Gradient Descents without bounded gradient assumption for the finite sum minimization , 2022, Neural Networks.

[8]  Wei Yuan,et al.  A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference , 2021, Applied Intelligence.

[9]  Danilo P. Mandic,et al.  Convergence of the RMSProp deep learning method with penalty for nonconvex optimization , 2021, Neural Networks.

[10]  Lam M. Nguyen,et al.  A hybrid stochastic optimization framework for composite nonconvex optimization , 2021, Math. Program..

[11]  Dongsheng Li,et al.  Novel Convergence Results of Adaptive Stochastic Gradient Descents , 2020, IEEE Transactions on Image Processing.

[12]  Tao Sun,et al.  Nonergodic Complexity of Proximal Inertial Gradient Descents , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Wotao Yin,et al.  An Improved Analysis of Stochastic Gradient Descent with Momentum , 2020, NeurIPS.

[14]  Michael I. Jordan,et al.  Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization , 2020, SIAM J. Math. Data Sci..

[15]  Mikael Johansson,et al.  Convergence of a Stochastic Gradient Method with Momentum for Nonsmooth Nonconvex Optimization , 2020, ICML.

[16]  Aytaç Altan,et al.  THE EFFECT OF KERNEL VALUES IN SUPPORT VECTOR MACHINE TO FORECASTING PERFORMANCE OF FINANCIAL TIME SERIES , 2019 .

[17]  Ke Tang,et al.  Stochastic Gradient Descent for Nonconvex Learning Without Bounded Gradient Assumptions , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Mark W. Schmidt,et al.  Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron , 2018, AISTATS.

[19]  Karthik Sridharan,et al.  Uniform Convergence of Gradients for Non-Convex Learning and Optimization , 2018, NeurIPS.

[20]  Ruoyu Sun,et al.  On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization , 2018, ICLR.

[21]  Xiaoxia Wu,et al.  AdaGrad stepsizes: Sharp convergence over nonconvex landscapes, from any initialization , 2018, ICML.

[22]  Francesco Orabona,et al.  On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes , 2018, AISTATS.

[23]  Shuai Li,et al.  A New Varying-Parameter Convergent-Differential Neural-Network for Solving Time-Varying Convex QP Problem Constrained by Linear-Equality , 2018, IEEE Transactions on Automatic Control.

[24]  Peter Richtárik,et al.  SGD and Hogwild! Convergence Without the Bounded Gradients Assumption , 2018, ICML.

[25]  Zhijun Zhang,et al.  A New Varying-Parameter Recurrent Neural-Network for Online Solution of Time-Varying Sylvester Equation , 2018, IEEE Transactions on Cybernetics.

[26]  Zhijun Zhang,et al.  A Varying-Parameter Convergent-Differential Neural Network for Solving Joint-Angular-Drift Problems of Redundant Robot Manipulators , 2018, IEEE/ASME Transactions on Mechatronics.

[27]  Mark W. Schmidt,et al.  Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition , 2016, ECML/PKDD.

[28]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[29]  Saeed Ghadimi,et al.  Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming , 2013, SIAM J. Optim..

[30]  Ohad Shamir,et al.  Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes , 2012, ICML.

[31]  John C. Duchi,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011 .

[32]  John N. Tsitsiklis,et al.  Gradient Convergence in Gradient methods with Errors , 1999, SIAM J. Optim..

[33]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[34]  Aaron Defazio,et al.  Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ball , 2021, COLT.

[35]  Yingbin Liang,et al.  SpiderBoost and Momentum: Faster Variance Reduction Algorithms , 2019, NeurIPS.

[36]  Alexei A. Gaivoronski,et al.  Convergence properties of backpropagation for neural nets via theory of stochastic gradient methods. Part 1 , 1994 .

[37]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[38]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .