论文信息 - A Parallel SGD method with Strong Convergence

A Parallel SGD method with Strong Convergence

This paper proposes a novel parallel stochastic gradient descent (SGD) method that is obtained by applying parallel sets of SGD iterations (each set operating on one node using the data residing in it) for finding the direction in each iteration of a batch descent method. The method has strong convergence properties. Experiments on datasets with high dimensional feature spaces show the value of this method.

S. Sundararajan | S. Sathiya Keerthi | Léon Bottou | Dhruv Kumar Mahajan

[1] Chih-Jen Lin,et al. Trust Region Newton Method for Logistic Regression , 2008, J. Mach. Learn. Res..

[2] Chih-Jen Lin,et al. A dual coordinate descent method for large-scale linear SVM , 2008, ICML '08.

[3] Léon Bottou,et al. Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[4] Tong Zhang,et al. Accelerating Stochastic Gradient Descent using Predictive Variance Reduction , 2013, NIPS.

[5] Gideon S. Mann,et al. Efficient Large-Scale Distributed Training of Conditional Maximum Entropy Models , 2009, NIPS.

[6] Mark W. Schmidt,et al. A Stochastic Gradient Method with an Exponential Convergence Rate for Strongly-Convex Optimization with Finite Training Sets , 2012, ArXiv.

[7] Stephen P. Boyd,et al. Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[8] Alexander J. Smola,et al. Parallelized Stochastic Gradient Descent , 2010, NIPS.

[9] Thomas Hofmann,et al. Map-Reduce for Machine Learning on Multicore , 2007 .

[10] John Langford,et al. A reliable effective terascale linear learning system , 2011, J. Mach. Learn. Res..

[11] Gideon S. Mann,et al. MapReduce/Bigtable for Distributed Optimization , 2010 .

[12] Kunle Olukotun,et al. Map-Reduce for Machine Learning on Multicore , 2006, NIPS.

[13] Mark W. Schmidt,et al. A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets , 2012, NIPS.