Distributed mini-batch random projection algorithms for reduced communication overhead

We propose a gossip-based mini-batch random projection (GMRP) algorithm that can reduce communication overhead for a distributed optimization problem defined over a network with a very large number of constraints. We state a convergence result and provide an application of the GMRP, text classification with support vector machines.

[1]  Angelia Nedic,et al.  Random projection algorithms for convex set intersection problems , 2010, 49th IEEE Conference on Decision and Control (CDC).

[2]  John Langford,et al.  Scaling up machine learning: parallel and distributed approaches , 2011, KDD '11 Tutorials.

[3]  Asuman Ozdaglar,et al.  Cooperative distributed multi-agent optimization , 2010, Convex Optimization in Signal Processing and Communications.

[4]  Ali H. Sayed,et al.  Diffusion Adaptation Strategies for Distributed Optimization and Learning Over Networks , 2011, IEEE Transactions on Signal Processing.

[5]  J. Burkey,et al.  WEAK SHARP MINIMA IN MATHEMATICAL PROGRAMMING , 1993 .

[6]  Eduardo F. Camacho,et al.  Randomized Strategies for Probabilistic Solutions of Uncertain Feasibility and Optimization Problems , 2009, IEEE Transactions on Automatic Control.

[7]  Angelia Nedic,et al.  Distributed Random Projection Algorithm for Convex Optimization , 2012, IEEE Journal of Selected Topics in Signal Processing.

[8]  Jean-Philippe Vial,et al.  Robust Optimization , 2021, ICORES.

[9]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[10]  Bruce H. Krogh,et al.  Distributed model predictive control , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[11]  Stephen P. Boyd,et al.  Randomized gossip algorithms , 2006, IEEE Transactions on Information Theory.

[12]  Soummya Kar,et al.  Distributed Consensus Algorithms in Sensor Networks: Quantized Data and Random Link Failures , 2007, IEEE Transactions on Signal Processing.

[13]  Boris Polyak.  Random Algorithms for Solving Convex Inequalities , 2001 .

[14]  Angelia Nedic,et al.  Random algorithms for convex minimization problems , 2011, Math. Program..

[15]  Giuseppe Carlo Calafiore,et al.  Random Convex Programs , 2010, SIAM J. Optim..

[16]  Thorsten Joachims,et al.  Training linear SVMs in linear time , 2006, KDD '06.

[17]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.