Exact asymptotic characterisation of running time for approximate gradient descent on random graphs

In this work we study the time complexity for the search of local minima in random graphs whose vertices have i.i.d. cost values. We show that, for Erdös-Rényi graphs with connection probability given by λ/nα (with λ > 0 and 0 < α < 1), a family of local algorithms that approximate a gradient descent find local minima faster than the full gradient descent. Furthermore, we find a probabilistic representation for the running time of these algorithms leading to asymptotic estimates of the mean running times.

[1]  Yuan Yao,et al.  Global Convergence of Block Coordinate Descent in Deep Learning , 2018, ICML.

[2]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[3]  R. Shorrock.  On record values and record times , 1972, Journal of Applied Probability.

[4]  Lena Schwartz.  Record Values Theory And Applications , 2016 .

[5]  Léon Bottou,et al.  The Tradeoffs of Large Scale Learning , 2007, NIPS.

[6]  S. Crawford,et al.  Volume 1 , 2012, Journal of Diabetes Investigation.

[7]  P. Ferrari,et al.  Independence of thinned processes characterizes the Poisson process: An elementary proof and a statistical application , 2007 .

[8]  John Wright,et al.  When Are Nonconvex Problems Not Scary? , 2015, ArXiv.

[9]  Stephen J. Wright.  Coordinate descent algorithms , 2015, Mathematical Programming.

[10]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[11]  R. Shorrock,et al.  Record values and inter-record times , 1973, Journal of Applied Probability.

[12]  Nicholas I. M. Gould,et al.  SIAM Journal on Optimization , 2012 .

[13]  Wotao Yin,et al.  Block Stochastic Gradient Iteration for Convex and Nonconvex Optimization , 2014, SIAM J. Optim..

[14]  D. R. Cox.  Journal of Applied Probability , 1964, Canadian Mathematical Bulletin.

[15]  Bruno Gaujal,et al.  Complexity and Optimality of the Best Response Algorithm in Random Potential Games , 2016, SAGT.

[16]  Yiguang Hong,et al.  Potential game design for a class of distributed optimisation problems , 2014 .

[17]  Editors , 2003 .